Arithmancy wasn’t just a class Hermione took at Hogwarts. It’s real math magic that can trick you into thinking you’re immune to covid, take a covid medicine that doesn’t work, and invest in a fake hedge fund. It does involve math, so if you’re like Ron, don’t even bother. 
A comet passes by the earth & someone tweets that it's given some 6 year olds the power to control coin flips with their minds. To see if it’s true, a child just needs to flip a coin heads many times in a row. But how many is enough to prove she has powers?
Well, it can’t just be 3 times, because that would happen by chance 1 out of 8 times (just multiple 2x2x2), as 12.5% chance of a powerless child appearing to have powers. So some clinical scientists propose using what’s called a one-sided p<0.025 test. That just means…
…that the child must flip the coin heads enough times that chance of her having done so by pure luck is <2.5%. Well that would be 6x in a row, which has a 1/64 chance. Kids should send in videos of their consecutive flips. Turns out, in a village with 80 kids, one does it.
The local paper touts her accomplishment & declares that she’s a bona fide witch. Hogwarts starts drafting a recruitment letter. Across the country, about 60,000 excited kids have the power. But scientists say that this one test isn’t enough. It takes two trials to be sure.
So all the towns & villages stage a live viewing of all their magician kids flipping coins again. And they are shocked to discover that nearly all of them fail to flip six heads in row. In fact, from 64k kids, ~63k fail. But that still leaves ~1000 who do it. They are amazing!
These 1000 are now hailed as true witches and wizards, bestowed with cometly powers. These children have watched themselves flip heads 6x in a row, twice in a row. They certainly believe. They must vow not to use their powers to make coin flips favor one sports team over another.
Now let’s set aside silly magic coin flip powers & instead talk about drug development. Running a clinical trial is actually not so different from seeing whether a drug has special powers. In a covid trial, we see if a drug saves more lives than not using the drug.
In fact, to make sure doctors & patients don’t do anything different from knowing who gets drug, we “blind” the study by randomly giving patients either a drug or a placebo pill or injection. We then look for a difference in death rate big enough that it’s unlikely due to luck.
But “unlikely” is actually a number and it’s typically 2.5%. When you hear a clinical trial result as being “statistically significant”, that usually means that the odds of the drug appearing to work as well as it did compared to placebo was less than 2.5%, or 1 in 40.
Actually, they may talk about a p value being <0.05 & therefore it seems like cutoff is 5% (1 in 20), but that 5% is actually made of two sides… 2.5% that drug might look a lot better than placebo by chance & 2.5% that placebo might look way better than drug, which is a fail.
So really a two-sided p<0.05 really means a one-sided p<0.025, which is a 2.5% chance that the drug, if it doesn’t work at all, happens to look like it’s way better than placebo. You could set a different threshold. You could say that you’ll only trust results where...
...a drug's effect is so large that it could only have occurred by chance 1 in 100 times, or a one-sided p<0.01. Such thresholds are set by convention, and the most common threshold happens to be a one-sided p<0.025, or 2.5% which is a 1:40 chance.
So when a drug looks like it works that well, we say “Wow, odds are that drug really works because if it didn’t, the odds of appearing to work that well by chance alone are 1:40 so it couldn’t have been luck.” And yet, of course if could have been luck.
There are now over 400 Covid clinical drug trials ongoing. That means, even if none of those drugs actually work, then by chance alone, 10 of them will appear to have a statistically significant benefit. So let’s say that among 410 drugs being tested, 10 actually work.
It’s important to now that even if a drug really works, a clinical trial might not show it since there’s always bad luck that could make the placebo appear to work better or the drug appear to work worse. E.g. a child who actually can make a coin flip heads 75% of the time…
…still has a 12.5% chance of flipping tails twice in a row. Trials are often designed to have an 80% chance of demonstrating that a drug works if it actually works as well as the scientists believe it does (and only a 2.5% chance of looking like it works if it actually doesn’t).
That means that out of 410 drugs of which 10 drugs really work, 18 drug trials would be statistically significantly positive: 8 that really work and 10 false-positives. So if you had covid & took one of those drugs, you would have only a 44% chance of getting an effective drug.
Maybe you see parallels here to antibody tests that tell us who's actually recovered from covid. I’ve colorfully explained how such tests can be misunderstood & trick people into thinking they are immune when they aren’t.
But drugs usually are not approved based on results of one clinical trial. The FDA typically requires two. Which means if we retested the 18 drugs that were positive in their first trial, then we would very likely see that the 10 that are truly effective fail their 2nd trial.
And of the 8 that really are effective, 6 achieve a statistically significant result in their second trial and therefore stand out as clearly effective. There are a lot of caveats that I won’t go into but I’ll list them here for experts who object to my simplifications.
Some trials have more than one endpoint and therefore more than one way to win, which increases chance of them looking positive by dumb luck. Failed trials often lead to repeated attempts, which, like losing at ping pong and yelling “best out of three” and then losing again…
…and yelling “best out of five” (my younger brother would do that), increases odds of ineffective drug appearing to work. Especially when the first trial works (think of those 60,000 kids who flipped heads 6x in a row the first time), it’s hard to accept failure the second time.
Amidst this pandemic, we are rushing drugs to patients. There may be some drugs that appear to work well in a trial, but it’s going to be important to repeat that trial to cut down on the false-positives. What helps is not to test drugs that make no sense to test.
For example, it would be silly to run every single one of Bertie Bott’s Every Flavour Beans through its own Covid clinical trial. One might appear to work, but only due to chance (even though to the scientists backing the earwax bean…
…it would seem like their hypothesis really panned out). Similarly, there are some drugs being tested that have a flimsy “mechanistic rationale”. There are >40 of them, I’m sure, & one of them is probably going to appear to work & yet that will be due to chance.
Our analyst team (I’m an investor and RA Capital invests in biotechnology) is tracking many Covid drugs in development using a technology landscape mind-map. We teach on this map which drugs have a stronger rationale for working. racap.com/covid-19
Therefore, if one of these drug trials is positive, it’s more likely to be true. For example, Gilead’s remdesivir is actually designed to inhibit certain viral proteins and works in the laboratory to shut down SARS-CoV-2. So when it worked in a clinical trial, that was credible.
There are other drugs that are similar to Remdesivir which make sense to test in clinical trials. There are also antibodies that mimic what how the immune system combats viruses. Those also make sense. Unfortunately, continuing to test drugs like HCQ makes less sense.
The greatest harm from testing nonsense drugs during a pandemic is they absorb patients into trials who could be better allocated to trials of drugs that are more likely to work. Instead of quickly enrolling 200 patients into a trial of a remdesivir-like drug or an antibody…
…those 200 patients are spread thinly across many other trials of drugs whose effectiveness is implausible or even disproven. It therefore takes us longer to get useful answers. We’re seeing timelines get delayed for getting clinical data from many trials on our Covid map.
While drug development can seem like a creative brainstorm where there are no bad ideas, that’s not actually the case. There are bad ideas. And those bad ideas cause harm, especially in a pandemic, by distracting resources from the good ideas.
Some might object “who is to say what's a good or bad idea? That’s what clinical trials are for!” but I think clinical trials should be used for testing good ideas. There are smart people out there who can analyze preclinical data to separate the good from bad ideas.
The trouble is that those smart people are sometimes wrong. & whenever they are caught being wrong, e.g. saying something won’t work & then it does, that mistake is used to discredit them, as if to say that if no one is 100% accurate, then everyone is equally likely to be right.
And so we have what appears to be this democracy of scientific ideas, crowding clinical trials, slowing development of scientifically more-worthy drug candidates. But if some of those improbably projects do appear to work, remember arithmancy and coin-flipping kids.
We’ll need to repeat trials to weed out the false-positives. But testing long shots isn’t always a bad idea. Sometimes we learn radically new things when improbable ideas pan out. It’s just that when time is short amidst a pandemic, we must triage & focus on the best ideas.
Now, didn’t I also tell you that arithmancy could also trick you into investing in a fake hedge fund? Yes… indeed. So here’s a story of how that might happen. It was told me years ago by someone who seemed like he had given it a lot of thought.
A smooth-talking would-be fund manager separately tells 1024 wealthy individuals that he’s starting up a hedge fund. He says he’ll offer each of them 6 correct stock tips to prove his skills & will then let them invest with him. He then tells 512 that AMZN will go up next week…
…& to other 512 he says that AMZN will go down. After 1 week, AMZN has gone up a few percent & the 512 who saw him get it right call back for a 2nd tip (the other 512 realize he’s a fraud). For his 2nd tip, he half (256) that FB will go up next week & half that it will go down.
FB happens to go down and another 256 realize he’s a fraud. He repeats this with GOOG, AMGN, & TSLA, wittling down his fan base to 128, 64, and 32. So now he has 32 investors who have seen him trade 5 stocks correctly in a row. He does the same trick with APPL.
So now 16 investors have seen him get 6 trades right in a row. They each offer him $10M and he now has a $160M hedge fund. Since he doesn’t actually know how to invest, he diversifies his holdings across many different companies.
Although his portfolio tracks the market closely, his investors’ returns are below average b/c of his hefty fees. He makes all kinds of excuses, but after a few years, they realize that he’s not a good investor & take back their money, but not after the trickster's made millions.
Note that the hardest thing about this whole strategy is getting to know 1024 wealthy people capable of investing $10M in a fund. The need to start with large numbers is what makes such a con unlikely.
Yet people can also trick themselves into believing they’re good investors. Imagine 1024 people who start to trade stocks. 16 will get 6 trades right in a row. They will think they are awesome investors. 8 will get the 7th right, 4 an eighth, 2 a 9th, & 1 will get 10 in a row!
Watching your own luck happen like that is a total mindbender. It’s like the proverbial million monkeys with typewriters clanking away. One of them eventually bangs out Hamlet. To us, we know it’s chance. But to the monkey! It thinks it’s Shakespeare reincarnated.
It’s very hard to accept that one’s own success could have been due to chance. And yet, one has to always accept that there’s a chance. Therefore, a good scientist and a good investor will always ask themselves… “does it make sense that this worked? Can I repeat it?”
In fact, the way to be more confident that you’ve really recovered from Covid if a test says you have antibodies is to get retested using a different antibody test. If that’s also positive, then odds are better that you really had it.
The less likely you were to have had it (e.g. your town hasn’t had a case in weeks), the more likely that your first test was a false-positive and the more important it is for you to get retested.
The less training you have at investment analysis, the more important it is to be skeptical of your initial success investing (and even then, always be skeptical. The market is always learning to be smarter than you. My partner and I constantly question ourselves).
The scientific community is currently running millions of experiments of all kinds every year and over 60,000 clinical studies at a time. There are many false-positives. Scientists must accept that it’s always possible that they are the lucky monkey, the lucky 6 year old…
They must remain skeptical of their findings and repeat experiments several times to become more confident. There’s a 1 in 1600 chance of winning 1 in 40 odds twice in a row. That’s rare, but with so many trials out there, repeating a third time isn’t a bad idea.
And especially when we’re not talking about stocks or coin flips, but saving lives, it’s important that the bad ideas get out of the way so that patients can be enrolled in the clinical trials of drugs that have the most plausible chance of actually saving lives.
For Frequently Asked Covid Questions and for covid explainers of all sorts, please see my pinned tweet.
And if you have just a bit of extra time on your hands and want to know about the math behind drug pricing and what's really at the heart of why patients struggle with affording medicines, I wrote a book that has plenty of analogies you will enjoy. thegreatamericandrugdeal.com
If you made it this far & believe world would be a little better and saner if more people were vaccinated against being fooled by arithmancy, then join in the power of exponential education by tweeting this thread out. Here’s a shortcut to the beginning.
