Thread by @callin_bull

A couple of months ago, an almost unfathomably bad paper was published in the Journal of Public Health: From Theory to Practice.

It purports to prove—mathematically—that homeopathy will provide and effective treatment for COVID-19.

link.springer.com/article/10.100…

While it would be fish in a barrel to drag this paper as a contribution to the pseudoscience of homeopathy, we'll largely pass on that here. More interestingly, this single paper illustrates quite a few of the points that we make in our forthcoming book.

The first of them pertains to the role of peer review as guarantor of scientific accuracy.

In short, it's no guarantee, as we discuss here: callingbullshit.org/tools/tools_le…

This paper shows that all sorts of stuff makes it through peer review.

It's better not to think about peer review as guaranteeing that a paper is right. It's more like a process of enriching the final pool of published papers for ones that are accurate, and interesting. Peer review attempts, to varying degrees, to sort out that that are not.

Second there's the issue of predatory journals versus journals from "reputable" publishers. What's a predatory journal? Peter Burns has a nice data graphic (below; link to original library.stonybrook.edu/wp-content/upl…) and we discuss this in detail in the article linked above.

The point is that the Journal of Public Health NOT a predatory journal in any conventional sense of the term. It's published by Springer, one of the "big five" academic publishers (journals.plos.org/plosone/articl…). In principle, @SpringerNature has every reason to protect its reputation.

Yet the paper is absolute bullshit—and the reasons why are quite interesting to unpack.

For starters, the paper has nothing to do with COVID. We get a gives a cursory overview of the pandemic in the introduction, and that's it.

Mason Porter pretty much has their number.

That should be a red flag. But where the paper gets really wild is in its mathematical "proof" of the principles of homeopathy.

There is no way we can adequately unpacking everything here.

What follows is a weak attempt; we hope to pass the baton to others of greater skill.

To put all of this into context, our book is basically about the way that people bullshit using quantitative language, whether it is figures and statistics, machine learning algorithms, or mathematical formalism.

Few of us feel qualified to challenge information of this sort.

Few of us have the training to be able to cogently address the technical issues about a multiple logistic regression or a random forest classifier. And in any event, the world of numbers seems to be a world of objective facts that come straight from nature. Who can question that?

The answer is that we all can. And the book explains why—and how.

But back to this paper.

To understand it, you have to understand the concept of a *prover* in homeopathy.

So at the risk of a very brief digression, a prover is a test subject who is willing to ingest a potential treatment to determine its physiological effects.

Why does that matter? It's because the fundamental principle of homeopathy is *similia similibus curentur*—i.e, "like cures like".

That which causes symptoms in healthy person, when presented in properly diluted form, cures those same symptoms in one who is ill.

Incidentally, giving something a Latin name is not dissimilar to dressing it up in the language of mathematics. It confers an unwarranted sense of authority while leaving the common person-on-the-street feeling unqualified to challenge its authority.

Ok, so that's a prover. On to the mathematics. The authors assign letters to the patient and the prover.

What do these letters correspond to, precisely?

Who cares! They're math.

We then get a long string of definitions. I may want to refer to these later, so I quote them in full.

What is striking about all of this is the level of abstraction. We are given no idea about what form any of these quantities take.

Sometimes, mathematical abstraction is a sign of depth, sophistication, and generality.

Other times, it's an indication that someone is selling you horse meat dressed up as filet mignon.

Let's look at the mathematical proof.

The authors take chi_0 as some sort of representation—of unspecified form—of the "biochemical kinetics" of the individual.

(FWIW, the term chi is replaced with the term x shortly after it is introduced. I guess they look pretty similar.)

I'll use "x" rather than "chi" going forward.

A is some sort of operator—also of unspecified form—that when applied to the original state x_0 yields the diseased state x_i.

C is another operator that does the same, but for the action of the proposed treatment on the "prover".

Ok, now it gets sketchy. We are now given an operator B that somehow transforms between patient and prover. It works in both normal and perturbed states.

(We also get a therefore that doesn't follow from the preceding, but is just a repeat of a previous definition.)

Substituting our definition of B into our definition of C, we get the relationship which I suppose in words means something like "the perturbed state of the prover is equal to the perturbed state of the patient, once we use magical transformation B on it."

"Then we invert B."

"What?"

"You heard me, we invert B."

"What if it's not invertible?"

"Of course it's invertible. We haven't specified WTF it is. From this line, though, it's clearly an invertible something-or-other."

So far it's all been definitions.

Now buckle up, because here in the next couple of images comes *all of the actual math in the paper*.

This says that if we magically transform prover into patient, then apply the disease, then magically transform back into the prover, we get the prover after perturbing with the treatment.

This is just a restatement of the assumptions of the model. But damnit it looks mathy now!

And here, the pièce de résistance:

We've already defined C as the effect of the treatment on the prover. So we can substitute.

What this says is that the effect of the treatment on the prover is the effect of the disease on the patient, if we pass it forward and backward through magical operator B that somehow interconverts prover and patient in both perturbed and unperturbed state.

That's it.

That's the whole paper.

You might want to slow down and go through it again.

I'll wait.

Higher math takes time.

Suddenly the nature of the operators is revealed.

They're affine transformations now. A bit late, but better than never.

Affine transformations. That's some serious math talk.

If you don't remember what an affine transformation is, you don't have any right to criticize us!

(BTW, affine transformations aren't so hard at all. They're just linear transformations—which @3blue1brown explains far better than I could here—, plus a translation.)

Of course nowhere have the authors done anything mathematically to work with the fact that these are affine transformations. It's just big words getting thrown around to no purpose.

And there's the invertibility issue as well, but leave that under the rug where it belongs.

And so we reach the end of our story. It might look like we have proven a parallelism between the action of treatment on prover and the action of disease on patient.

But of course we haven't. We've merely assumed it, then restated it using capital letters and an equals sign.

What we've just gone through is the epitome of pseudomathematical bullshit.

In our book, we define bullshit thusly:

Bullshit involves language, statistical figures, data graphics, and other forms of presentation intended to persuade by impressing and overwhelming a reader or listener, with a blatant disregard for truth and logical coherence.

This is exactly what the authors have done here. The mathematical notation serves to impress and/or overwhelm a reader, so that the conclusions seem unchallengeable—and the whole thing is entirely lacking in logical coherence.

It seems to have impressed and/or overwhelmed the editor and reviewers, in any case.

Now we have no idea whether this is deliberate deception on the part of the authors or not. We're less concerned about that than Harry Frankfurt (of _On Bullshit_ fame) would be. For Frankfurt, bullshit is in the mind of the bullshitter; it requires intentionality.

We are sympathetic to G. A. Cohen, who situates bullshit in the mind of the beholder. For Cohen, the consequences, not the intentions, are what matter.

Here the consequence is to obfuscate using a caricature of mathematical notation.

learning.hccs.edu/faculty/robert…

What we find so striking about this example is that it's so poorly done.

It is hard to understand how anyone could consider this rewriting of ill-formed definitions to constitute scientific progress.

But if one wanted to deceive, it would be so easy to doing much better.

We're probably taking this way too seriously, but there's another thing we don't understand.

The purpose of treatment we thought would have been to restore normal physiology: In the language of this paper, to restore x_i to x_0.

Presumably one does that using treatment C.

But the paper doesn't show that C does this.

The central result, if you can call it that, is that C screws up the prover exactly the same way that the disease screws up the patient. C=BAB^-1

Maybe we just don't understand the principles of homeopathy, but what a disappointment!

Getting serious for one minute: We request an explanation from @SpringerNature / @SpringerPublicHealth for how something like this could possibly make it into print, especially given the potential for severe harm to patients should anyone be foolish enough to take it seriously.

We urge the editorial board of the journal, below, to take the appropriate steps: consider whether retraction is warranted, reexamine the internal processes that allowed this to be published in the first place, and revise them to prevent this from happening again in the future.

Associate editors of the journal may wish to consider whether their time and expertise is well spent on venue that associates their names with work of this caliber.

We'd love to hear from Mike Daube whether his inclusion on the editorial board is with his permission. Has famously uncovered the low standards of other journals in this field.

https://twitter.com/TheEconomist/status/1266195540141842432

In the comments at @RetractionWatch, a far more concise expression of what we were trying to say about the mathematics here. retractionwatch.com/2020/07/25/pap…

A reader makes a very good point about this step. If we assume that A, B, C are linear transformations, this step is false. If we assume that they are more general operators, it is still false because it is false for linear transformations.

https://twitter.com/bernieb53293225/status/1287681384438521857?s=21

https://twitter.com/bernieb53293225/status/1287681384438521857

And that was quick. The Editor-in-Chief is now pursuing retraction.

retractionwatch.com/2020/07/27/bla…

The “overflow of COVID papers” excuse is perhaps honest but not remotely exculpatory in our opinion.

“There’s a massive pandemic that makes our journal super-relevant to life and death for billions of people, so naturally we dropped our editorial standards.”