Thread by @iowahawkblog

If you mean my old world where peer reviewers for the 2 leading medical journals in the world pause to check on the validity of an out-of-nowhere 3rd party data source before hitting "publish," you probably have a point

https://twitter.com/ajayspec2020/status/1268764300035923968

We really need to have an talk about the visceral reactions to the Lancet/NEJM retraction saga. I'm sorry if I've laid off the fart jokes for a few days to burrow down on this, but it's important and says a lot about how politics has utterly welded us to our priors.

On the one hand, there are people who are so doggedly committed to the proposition that Dr Drumpf's Snake Oil Suicide Pills created mountains of corpses around the world, no retraction or evidence to the contrary will dissuade them.

I would also note these are generally the people who LoVe sCIeNcE </jerkoff motion> while checking their horoscopes.

on the other hand, there are people who are so committed to the proposition that hydrochloroquine is the secret miracle cure that the globalist pharma bandits will stop at nothing to hide from you so the Chicoms can take over, nothing will shake them from that belief.

I would add these are generally the people who, upon learning of the Lancet retraction, immediately began dancing around like Homer Simpson, chanting "I AM SMART, S-M-R-T" while they set their house on fire.

I've spent a couple of days dragging the Lancet & NEJM for their gullibility and lack of skepticism, which was undoubtedly driven in part by too-good-to-check political implications.

But I don't want to burn down research journals, I want them reformed.

A good start would be moving to an open source review process. Let's face it, Lancet's editor and reviewers were utterly (and maybe willfully) asleep here, and it took some online statisticians and sleuths to ring the alarm bells on what was obviously a sketchy data source.

And while they're at it, maybe the Lancet and NEJM should politely inform the reviewers who approved publication of the studies based on Surgisphere data that their services will no longer be required.

*a primer on peer review:

1. Authors submit study to journal editor
2. Editor disguises authorship
3. Editor sends to initially anonymous reviewers (blind review)
4. Reviewer return with comments/criticisms/recs
5. Editor decides publication/revision/rejection

The reviewers here aren't like a commercial media outlet, they are other research academics whom the editor has identified as experts in the topic of the submitted paper.

Nobody here gets paid; neither the authors, reviewers, or editors. The payoff is prestige / tenure.

It's supposed to work well in theory; in practice not so much. "Blind" review is not so blind, and in the cozy world of academics it's probably not hard to guess whose paper you're reviewing. This can lead to log rolling - A reviews B's papers, and B reviews A's.

In any case, had the Lancet opened up the review process pre-publication, maybe someone would have thought to check on the extraordinary / fantastic claims of a worldwide COVID patient database, by a heretofore unknown 5-person data shop.

jfc, Surgisphere's one page website should have been enough to set off somebody's Spidey-tingles. Extra points for their claims of "advanced AI and Machine Learning," which these days is almost a calling card for charlatans.

*Not dismissing AI/ML, but there are a lot of startups who claim expertise in it to dazzle dopes, and this was clearly one of them.

That said, AI/ML is out of place in medical treatment research, which should be the purview of classical stats / experimental design.