1) If you achieve an observed effect in a (standard) study equal to the effect you powered for, the one-sided p-value is 0.0006. This has two implications, the first on avoiding “significant but meaningless” results, and the second on resource allocation (tweetorial)
2) Standard clinical trials, testing means and proportions with alpha=0.025, protect against bad luck. If you have 90% power for an effect X, you reject the null for anything above 0.604 X. This protects you against missing an effective therapy that was unlucky in your trial.
3) You should always ask your statistician “what is the smallest observed effect where I reject the null?”. If this value (0.604 X for standard trials, might be different in others) is clinically meaningless, you should rethink your experiment.
4) At the observed effect threshold for rejecting the null (0.604 X), the p-value is 0.025. If you get closer to an observed effect of X, the p-value is MUCH smaller, closer to 0.0006 (one sided). If your therapy truly has effect X, half the time you will get p<0.0006.
5) If effective therapies get p<0.0006 half the time, they usually achieve significance much earlier. This effect is qualitatively symmetric but more pronounced for futility. A null therapy typically has a 70-80% chance of being hopeless early.
6) Standard sample sizes are insurance against bad luck. I wouldn’t pay full replacement cost of my house for homeowners insurance, I pay less by sharing the risks. We waste resources in clinical trials my making all of them pay a sample size that is only needed occasionally.
7) Early futility stopping, and to a lesser degree early success stopping, allow us to share these risks across a portfolio. We might fund up to 50% more trials. More details on the NHLBI ICTR blog
innovativeclinicaltrial.org/hypothesis-tes…
innovativeclinicaltrial.org/hypothesis-tes…
