Thread by @GidMK

Some thoughts on scientific retractions. I think we can fairly say that there are far fewer than we'd expect if science was working well

There have been in the range of 100,000 scientific papers published on COVID-19 this year. PubMed shows 78,000, and if we include journals not indexed by that resource I'd imagine we'd break 100k easily

Now, let's think about this scientifically. What's the serious error/fraud rate for published research? The rate at which a retraction-worthy paper is published in a peer-reviewed journal

(Note, this obviously cannot be 0% because some papers ARE retracted)

To clarify, the rate that you think papers are published with errors that SHOULD lead to retraction, even if they are never retracted

https://twitter.com/GregHerczeg/status/1338602871697403905?s=20

So the results are rolling in, and it doesn't look great for published research! A lot of people who think that 1 in 200 papers is worthy of retraction 👀👀👀

But let's be more conservative and say 1 in 1,000 papers is bad enough to be retracted

With ~100,000 papers published on COVID-19 this year, we'd expect about 100 retractions

How many COVID-19 papers do you think have been retracted in 2020 (roughly)?

Now, this number is hard to come by, but if we use the database from @RetractionWatch it looks like about 44 papers have been retracted (although some were just preprints) retractionwatch.com/retracted-coro…

Given that this is not perfectly comprehensive, despite the excellent work of @RetractionWatch, let's say somewhere around 50 is about right

So, given an error rate of 1 in 1,000, half the retractions we'd expect

If the error rate is higher (and it very well may be), this number drops sharply. If we think that 1 in 200 papers is bad enough to be retracted, then only 10% of the expected retractions are happening

Bottom line: we know that there is an error rate in published research

EVEN IF THIS IS LOW, retractions are much rarer than we'd expect if the system was working well

Note: while it's true to say that retractions often take time, it's also true that of the 50 retractions around half were preprints. And if we only see retractions years after the pandemic has passed, isn't that an indictment on the system in and of itself?