,
13 tweets,
7 min read
Read on Twitter
So, yesterday we, @avigotsky and @TenanATC, put up a preprint about a new statistics technique in sport and exercise science. Today, I want to expand upon my thoughts and why I think it is emblematic of a greater problem in our field.
THREAD.
THREAD.
I am always skeptical of techniques to determine if “responders” exist. First, if you believe there is a physiological reason for responders then you should design an experiment to measure that heterogeneity (e.g., cross-over replicate HT @stephensenn doi.org/10.1177/096228…).
Second, if you are doing this post hoc, especially in a parallel group design, I honestly don’t know of a way to determine if the increased variance is from responders. The source of increased variance could be from many things (again HT to Senn doi.org/10.1002/sim.67… pg 968).
Moreover, we, as scientists, should seek out falsifiable hypotheses (I’m sure some science philosophy folks are howling reading this!). Simply saying “responders might exist” without ruling out other possibilities, and without theoretical explanation just simply isn’t good enough
Often when I see these analyses published they are tacked on as a way to explain why ‘significance’ wasn’t achieved. This is borderline pseudo-scientific (see ‘design’ section of my review doi.org/10.1080/233289…).
Now onto the math/stats. I won’t rehash to much of what Andrew did yesterday, but I will say this: this technique does not work as intended and will likely lead to many erroneous conclusions if implemented. It has problems similar to another technique used in my field…
That brings me to this point. My degrees are all in kinesiology/exercise science, but I did complete a graduate certificate in statistics and research methods (~24 credit hours). I tend to consider myself an applied statistician (that may even be too bold).
This means I APPLY what STATISTICIANS have clearly laid out as the best approach to answer my research questions. Therefore, *never in a million years* would I try to create a "novel" stats method and try to tell people they should use it (not w/o a math stats person involved)
IMHO, we’ve become flippant. I commonly hear “Don’t let stats interfere with a good story” far too much. We need listen to actual statisticians. Exercise scientists that are passionate about stats should collaborate with them and pass on their knowledge to the field.
If we continue down this path of using unreliable and inefficient stats methods we will tarnish the field’s credibility. If we simply use our numbers as “rhetorical tools” to show “what we already know to be true”... then well then science just turns into this…
So, where do we go from here? We ALL should collaborate, discuss, implement changes to make our science better. This goes beyond just statistics! There are brilliant people in this field and I want to see them shine.
I’m continuing to work with people on ways to improve stats practices within the field and communicate this to the scientists using them in practice. Want to help out? Join @STORKinesiology. There are other issues beyond stats and we need more people dedicated to addressing them.
@STORKinesiology @threadreaderapp unroll please
