I was going to write up a response to this PNAS piece: Redish et al., “Reproducibility failures are essential to scientific inquiry.” pnas.org/content/115/20…
But I realized that 1) PNAS might not publish it; 2) Even if they did, it would take a long time; 3) Posting my thoughts on Twitter would probably reach more people (about 2,000 people minimum, and a lot more depending on retweets).
So here goes. I’ll start with a summary of what Redish et al. argue. They say that “current discussions of the reproducibility crisis overlook the essential role that failures of reproducibility play in scientific inquiry.”
Again: “decades-long process of metabolizing reproducibility failures through theoretical integration is what leads to the reliable results across the sciences that have provided us with remarkable life-changing medical and engineering consequences.”
What do they mean? They have three examples from computer science and math.
In all three cases, someone proposed some finding/theory. Other people “failed to replicate” it in some context. After some debate, new discoveries come along that update the original finding/theory.
In all three cases, someone proposed some finding/theory. Other people “failed to replicate” it in some context. After some debate, new discoveries come along that update the original finding/theory.
Voila, science advances.
Fair enough.
Fair enough.
And when replication failures do indeed derive from “failures to generalize across what researchers hoped were inconsequential changes in background assumptions or experimental conditions,” then figuring out those factors can indeed drive scientific progress. I agree.
So what’s the problem?
FIRST, they say that "dissemination of this perspective to researchers, research funders, and the general public could positively influence the trajectory of ... scientific practice by preventing overzealous negative responses to the perceived reproducibility crisis . . .”
Whoa, there. Overzealous negative responses? What are those, exactly? No examples are given.
Indeed, one of the most typical “responses” to the reproducibility crisis (see Brian Nosek’s many talks) is that the entire research workflow should be more open and transparent (including analysis plan, methods, data, & code).
Far from being overzealous or negative, this idea would *directly attack* Redish et al.’s main problem: the fact that failures to replicate often arise because no one knows about the seemingly “inconsequential . . . experimental conditions.”
More transparency would help science advance faster by the very criteria that Redish et al. use.
SECOND, if anyone has sounded “overzealous” or “negative” in talking about reproducibility problems, it is because of the well-documented problems with research practices in many fields. I could provide literally hundreds of citations here.
The problem is not reproducibility failures arising from the mundane fact that any theory/finding is incomplete and needs to be updated over time. The problem is reproducibility failures arising from bad research practices coupled with publication bias favoring them.
But not just that. There’s a deeper meta-problem:
It’s easy, even trivially easy, for people who use bad research practices to defend themselves on Redish et al.’s theory.
It’s easy, even trivially easy, for people who use bad research practices to defend themselves on Redish et al.’s theory.
Whenever there’s a failure to replicate a study, the original author or his/her defenders point out that science is terribly difficult, that there are all kinds of moderating factors that may be hidden or unknowable, and that replications can fail for innumerable reasons.
See, e.g., Jason Mitchell’s (in)famous essay from 2014: jasonmitchell.fas.harvard.edu/Papers/Mitchel…
And all of this is true! Science *really is* hard. There *really are* lots of reasons that a replication could “fail.”
But the insidious problem here is that bad research practices (such as finding a positive result from exploiting one’s flexibility in how to clean or analyze data) are *easily disguised* as the “dynamic, iterative . . . process of discovery” that Redish et al. praise.
“Didn’t find the same result I found in my famous study that led to my TED talk? Oh, of course, you did the study in German undergraduates rather than Massachusetts undergraduates. Just one of those ‘inconsequential’ differences that lead to scientific advances!”
(Never mind that I p-hacked my study to death.)
In the end, I agree with nearly everything that Redish et al. say.
But their reasoning can be used as a convenient disguise for bad research practices -- a problem that has been overwhelmingly documented in many fields, but that they manage to avoid discussing altogether.
But their reasoning can be used as a convenient disguise for bad research practices -- a problem that has been overwhelmingly documented in many fields, but that they manage to avoid discussing altogether.
