1️⃣ Five years ago, I suggested systematically Sokaling all peer-reviewed journals. To “Sokal” is, hereby, to attempt to publish clearly bogus papers to illustrate the brokenness of the academic publication process.
2️⃣ Today @HPluckrose, @ConceptualJames & @peterboghossian reported on the first multiple-Sokaling. They were successful in publishing nonsense in top-ranked journals (which comes as no surprise, but is great to have verified).
3️⃣ I had in mind a more ambitious project. Pointing out that “grievance studies” fields are mostly nonsense is shooting fish in a barrel. We know that, for instance, “cognitive neuroscience” is also largely bollocks: biorxiv.org/content/early/… 
4️⃣ It’s an objective fact that the peer review process isn’t working. We also know, from @BrianNosek’s recent work, that it *can* work, if reviewers are incentivized to make it work. Money is enough, apparently. How about reputation?
@BrianNosek 5️⃣ Also in 2013, I suggested that there should be substantial career rewards for successful Sokaling.
There should also be consequences for editors and reviewers that accept demonstrably and intentionally bogus papers.
There should also be consequences for editors and reviewers that accept demonstrably and intentionally bogus papers.
6️⃣ Making Sokaling a routine part of the academic process would go a long way towards fixing it, I think. A few dozen Sokalings are a good start to raise awareness, but if—say—5% of all submitted papers were Sokals, reviewers and editors would become much more careful.
7️⃣ How would this work in practice? A paper would have to be registered as a Sokaling before first submission, along with an explanation of what the author thinks is wrong with it. A cryptographic time-locked database could ensure honesty about this.
7️⃣½ [That is, everyone could publicly verify whether or not a paper was intended as a Sokal. Once it was accepted, the author could unlock the pre-registration; after a determinate time period had elapsed, it would automatically unlock in any case.]
8️⃣ Fraud—invented facts—are a different problem from nonsense. In science, publishing a paper based on false, made-up data would not generally count. Either you use real data and give a bogus interpretation, or you describe a worthless data-collection process (with fake data).
9️⃣ This proposal is not nice. Unfortunately, it is too late for nice. Many-to-most academic fields run on a go-along-to-get-along basis, and now have large negative net value as a result.
Some fields should simply end. I suggested that for nutrition: meaningness.com/nutrition-resi…
Some fields should simply end. I suggested that for nutrition: meaningness.com/nutrition-resi…
🔟 A tension here: academia is increasingly awful as a career. That drives away many of the best researchers. Reforms that add to the suffering risk making a bad situation even worse.
Currently, reviewing papers is unpaid scut-work. Not surprising not everyone does it well.
Currently, reviewing papers is unpaid scut-work. Not surprising not everyone does it well.
1️⃣1️⃣ If peer review has reputational risk—it will be public knowledge if you recommend accepting a deliberately bad paper—many people may decline the job. (Though, successful detection of a hoax should look good on your CV!)
Publishers might have to pay reviewers…
Publishers might have to pay reviewers…
1️⃣2️⃣ … and the total number of papers published might drop precipitously if reviewers were more reluctant to recommend publication.
That would be good. Everyone agrees there’s WAY too much stuff published under the current system. A 90% reduction would be great.
That would be good. Everyone agrees there’s WAY too much stuff published under the current system. A 90% reduction would be great.
1️⃣3️⃣ I’d suggest that every PhD student be required to perform at least one attempt at Sokaling as a graduation requirement.
Learning what should count as unacceptably bad research is a critical part of learning how to do it well. And of spotting the difference in the lit.
Learning what should count as unacceptably bad research is a critical part of learning how to do it well. And of spotting the difference in the lit.
1️⃣4️⃣ Initially, everyone would go for low-hanging fruit in Sokaling attempts: the easiest ways to get nonsense past reviewers.
Reviewers would quickly catch on to the simplest tricks… then subtler errors.
And I hope this would lead to a virtuous upward spiral of quality.
Reviewers would quickly catch on to the simplest tricks… then subtler errors.
And I hope this would lead to a virtuous upward spiral of quality.
1️⃣5️⃣ Three questions:
𝔸) Would this make academia more adversarial, and thereby even more awful?
𝔹) Can a system develop that is adequately resistant to gaming (Goodhart’s Law)?
ℂ) Is it realistic to imagine something like this could actually happen?
𝔸) Would this make academia more adversarial, and thereby even more awful?
𝔹) Can a system develop that is adequately resistant to gaming (Goodhart’s Law)?
ℂ) Is it realistic to imagine something like this could actually happen?
1️⃣6️⃣ I’d love to hear your opinions about 𝔸 & 𝔹 !
As for ℂ — the crisis in academia is now obvious to all. Things cannot go on as they are. Unusual action becomes possible in extremis.
Recent dramatic process reforms in social psychology are startling, and inspiring.
As for ℂ — the crisis in academia is now obvious to all. Things cannot go on as they are. Unusual action becomes possible in extremis.
Recent dramatic process reforms in social psychology are startling, and inspiring.
Industrial-scale Sokaling of cancer research: 304 papers!
Deliberately targeting junk journals—one of many problems.
♻️@hbdchick
science.sciencemag.org/content/342/61…
Deliberately targeting junk journals—one of many problems.
♻️@hbdchick
science.sciencemag.org/content/342/61…
