,
16 tweets,
3 min read
Read on Twitter
Hi @scottlemaire, I would like to issue a public expression of concern about a paper published recently in the Journal of Surgical Research, on which you are the editor in chief.
The authors badly misunderstand the concept of post-hoc power. They have been engaged in correspondence with a number of statisticians replying to a letter they published elsewhere (the Annals of Surgery) wherein several of us have been trying to correct their misunderstanding.
The specific problem with the paper published in Journal of Surgical Reserch is this: the authors have done a "study" wherein they found a selection of articles, removed all of those with "significant" differences, performed post-hoc power calculations using the observed...
...effect sizes of the remaining studies. This is a self-fulfilling prophecy: by definition, any study that failed to find a significant difference is going to appear to have low "post hoc" power.
This is not a real finding of any significance, because post-hoc power using the observed effect size is just a different way of presenting the p-value, as explained here: daniellakens.blogspot.com/2014/12/observ…
The problematic understanding of "post hoc power" is a long-standing and apparently still pervasive myth in the clinical world, despite many efforts from statisticians to explain this problem.
I would be very grateful for your attention to this matter, as this paper's entire premise is flawed and it probably should be retracted. Very happy to discuss if you are willing, by phone, email, or any way you like.
If it helps, what they've done is essentially akin to performing a study on surgical mortality by abstracting consecutive cases of a particular surgery, removing all patients that survived (studies that found significant differences)...
...analyzing just the remaining patients that died (studies that didn't find significant differences), and concluding that surgical mortality is high based on their results. Finding that "post hoc power was low in studies that didn't find significant differences" is meaningless
And further, they use this "finding" to support a number of incorrect statements throughout their discussion about the implications.
To be clear, the analysis performed by the authors tells us absolutely nothing about whether the original studies were sufficiently powered to detect clinically meaningful or relevant effects.
And I still can't believe they managed to "remove studies that found significant results" followed by "post-hoc power calculations" & conclude that studies were "routinely unable to reach the power standard of 0.8" without the problem falling into place for them at some points
Like, how did they not run into the forehead-slapping moment of "Oh...all the studies that have high 'observed' power are the ones that found significant differences...and all the ones that have low 'observed' power are the ones that DON'T find significant differences...waaaait"
This paper is also akin to a headline "Randomized clinical trials consistently fail to find efficacious treatments when you remove all the positive results"
And, letting papers like this INTO the literature perpetuates some really awful misunderstanding of these concepts, potentially breeding a future generation of researchers that will allow this flawed work to inform how they design studies
