Thread by @TenanATC

Exercise & Sport Sciences has a poor track record developing "Novel Statistical Methods" & a new one was recently published:
link.springer.com/article/10.100…
As a reviewer on this paper now working to have it retracted, here is my perspective on how these methods get popularized

[THREAD]

First, a table of contents:
1. Invited to review the manuscript & suggest rejection
2. Contacted by outside researchers because concerns a flawed paper was published in @SportsMedicineJ where I am on the EB
3. I contact EiC to determine why the flawed paper was published
...
2/23

4. Develop simulations and formal mathematics providing evidence that the published method is deeply flawed
5. Contact the Authors, Dankel & Loenneke, regarding our work asking if our simulations/math are wrong and for their evidence
...
3/23

6. Since authors provide ZERO evidence their method works or that our math/simulations are wrong, we ask them to retract and they refuse
7. I ask EiC to Editorially Retract paper & they ask us to write a Letter to the Editor
8. Flawed method is already cited

Here we go...

4/23

(1) I was asked to review the paper on 4/23/2019 and raise a number of concerns and provide citations:
1a. There are general issues with "responder analyses" that the statistics field have already addressed:
trialsjournal.biomedcentral.com/track/pdf/10.1…
tandfonline.com/doi/abs/10.119…

5/23

1b. I also note that this issue, has been extensively discussed in more informal postings by esteemed statisticians such as @f2harrell in his informative forum post:
discourse.datamethods.org/t/responder-an…

6/23

1c. Finally, in my private note to the Editor I state:
"...their proposed method is not effective, nor does it solve the actual problem of looking at responders vs. non-responders. They have also not done a good enough literature review outside of the exercise science"

7/23

(2) I consider the issue settled and hopefully their flawed method does not see the light of day. HOWEVER, a month or so later I get contacted by multiple researchers asking me if I've seen this method and if it underwent statistical review prior to publication.

8/23

(3) I contact the @SportsMedicineJ EiC on 7/8/2019 to inquire why it was published & state that we have simulations showing that the error rates of the "novel method" are wrong and nonlinear.

Here is the realistic simulation with minute ventilation:
osf.io/6esbv/

9/23

(3) I am informed that "other two reviewers were also experts in the area (both in terms of statistical expertise and a specific interest in the topic) and I accordingly considered their comments and recommendations to be equally credible..."

10/23

(3) "...When experts disagree... we tend to side with the majority view. Given those circumstances, therefore, and without intending any disrespect to your views and concerns, I do not feel there is a case for retracting the paper."

11/23

(3) This aforementioned statement is fine when Editors have papers on what causes fatigue or blood flow dynamics during exercise... things where there may be valid viewpoints on the same or different evidence.

HOWEVER, that view does not work with the math is wrong.

12/23

(4) I work with my excellent colleagues @avigotsky & @ExPhysStudent to develop a fully fleshed out mathematical framework showing that not only is the "novel method" from Dankel & Loenneke theoretically flawed, the math just doesn't work out.

osf.io/ab683/

13/23

(5) On 8/8/2019, we contact Dankel & Loenneke regarding our concerns with their paper and provide all of our simulations, formal mathematics and code. We ask if anything is wrong or we have misinterpreted their method.

14/23

(5) We did have something wrong in our simulations. The authors don't actually define what an ERROR is even though they claim a 5% error rate. In our conversation with the Authors, it becomes clear that this claimed 5% error rate is only for "false positive" errors.

15/23

(5) While we disagree that one should only consider "false positive" errors and not other types of errors, we adjust our simulations/math.

Their error rates are still VERY WRONG. Their error rates can actually be north of 60%!!!!!!

16/23

(6) While the authors provided us with statistically sounding rationale for why they think they have a 5% error rate, they provide ZERO math or simulations which demonstrate the method works as proposed.

Next, we try to extend an olive branch.

17/23

(6) We ask the authors to retract their original article and that we will co-author a new paper with them on responder analyses, ways to account for measurement error in statistically sound ways (e.g. Deming regression), and other topics they intended to "solve".

18/23

(6) The Authors reject our olive branch proposal and instead simply state:

"We spoke with a biostatistician who recommended that we do not retract the paper."

Which is a terrible and obscure Appeal to Authority argument fallacy.

Time to go nuclear bitches.

19/23

(7) All of this dialog has included the EiC of Sports Medicine so I, again, ask the Editors to retract the paper because they Authors of the paper cannot provide evidence a method works.

This is equivalent to fudging data in a paper. It should be retracted.

20/23

(7) The Journal Editors ask us to write a formal Letter to the Editor. Here it is:

osf.io/wj498/

We still want this paper retracted and out of circulation so researchers don't use an invalid method thinking that @SportsMedicineJ has endorsed it.

21/23

(8) You might think we're overreacting, but flawed methods like this perpetuate. For example, Dankel & Loenneke have already used and cited their flawed method in a paper published in @APNMjournal

nrcresearchpress.com/doi/abs/10.113…

Flawed methods can easily get out of hand.

22/23

FIN.

So that's where we are now. We want the Editors to the right thing & retract the paper. We want the authors to recognize that the method is flawed & retract the paper.
If those fail, we will continually alert the field & anyone using the method that the results are invalid

Request review & RT @KristinSainani @keith_lohse @zbinney_NFLinj @cbpolis @jamesheathers @STORKinesiology @JamesSteeleII