Charles M. Beasley, Jr and Roy Tamura: What We Know and Do Not Know by Conventional Statistical Standards About Whether a Drug Does or Does Not Cause a Specific Side Effect

Donald F. Klein’s comments

        Charles Beasley and Roy Tamura's treatise shows that  the statistically minded clinician knows that rare events are trouble   When it comes to the minuscule - 1/1000, therapeutic or toxic effects - enormous, impractical samples would be necessary for RCTs. We have given up hope of concluding from RCTs if It is there or not. Beasley and  Tamura have assiduously put numbers on this generalization but have not contradicted it. In keeping with the substantial literature on the detection of rare events, they show that big, impractical sample sizes are required for the lucky RCT experimenter to have a fair chance of coming to an accurate conclusion. They use a variety of modern statistical approaches and show that different sample size estimates occur, but the conclusions do not break out of impractical sample space.  I believe that the rapidly developing area of machine learning has not been applied but am not optimistic. My current opinion is that the RCT is too impractical to fulfil this goal.

        There are multiple confounded approaches to naturalistic data. The great benefit of randomization, which balances out the variables you don't know about, is not available. I suspect reasonable, if shaky, conclusions will require similar large sample sizes that may be available from Scandinavian archives.

August 8, 2019