Thread by @jasoncrawford

Fascinating story I ran across reading the history of polio:

In 1954, when it was time for large-scale human trials of the vaccine, some researchers were *against* the idea of doing a properly randomized, double-blind, placebo-controlled clinical trial—including Salk himself

What did they want to do instead? An “observed control” trial. They would ask for volunteers (children) to get the vaccine, and then compare the rate of polio in the volunteers to the rate in their schoolmates who weren't vaccinated.

No placebo. No randomization. Not blind.

Of course, this was hopelessly confounded. In that era, the families most likely to volunteer were the more educated and affluent families (and those were actually the ones *most* at risk for the disease).

So why did Salk and others oppose proper randomized blind controls?

The argument against a randomized trial was the urgency of protecting the nation's children against a debilitating and deadly disease. If the vaccine worked, it would be a tragedy to withhold it from the control children:

From *Polio: An American Story*, by David Oshinsky:

Some context: the trial was massive, involving hundreds of thousands (eventually over a million) children across the country. This is not n=50 we're talking about here.

And the disease was seasonal, striking in epidemic waves every summer. So even if all the controls were properly vaccinated at the end of the trial, it would be too late for anyone who had been stricken that year.

So there was a real dilemma here. Salk himself seemed to be already convinced that the vaccine worked, and wanted it to be administered as widely as possible. Quoting again from *Polio: An American Story*:

Ultimately multiple people resigned over the issue, including Joseph Bell, the scientific director who was to run the trials for the National Foundation for Infantile Paralysis (which was sponsoring them).

The Foundation replaced Bell with Thomas Francis (Salk's old mentor)…

But Francis *also* insisted on a proper blind placebo control! Salk ended up going along with it (perhaps because he had to, perhaps because he trusted Francis more than Bell?)

In the end they did a combination. Some counties did a placebo control, others an “observed control”, at their own discretion.

Fortunately, there were enough randomized controls to draw sound scientific conclusions at the end of the study.

While I'm sympathetic to the practical issue of wanting to protect patients, I think the history of medicine shows how easy it is for scientific “knowledge” to become polluted with falsehoods based on less-than-perfect experiments.

In medicine especially, these mistakes become tradition, entrenched “wisdom” from revered authority figures that can stand undefeated as common practice for decades or centuries.

So it's important to get things right, and I think Bell and Francis were clearly correct here.

Mostly I just find it fascinating that as late as the 1950s, the need for proper randomized blind placebo controls in clinical trials was not universally accepted, even among scientific researchers.

Cultural norms matter, especially epistemic norms.

Turned this thread into a blog post:

https://twitter.com/rootsofprogress/status/1208107690993606658

PS: Some followup to the replies.

First, I'm very sympathetic to the dilemma that Salk faced. I think it's a tough problem, and it's worth thinking about different ways to approach it. I didn't mean to cast aspersions on Salk.

One way in general to improve this situation is to make sure that all the controls get the treatment immediately after the trial, if it is proved safe & effective. But—

In this case that wouldn't have changed anything. Polio was a seasonal disease, peaking each summer. And in the '50s, doing all the data analysis took several months. So getting the vaccine after the trial meant getting it the next season.

Some people have suggested that trials can be ended early if the data clearly shows a conclusion. This is true, although it's trickier than it appears—if you do it in a naive way, you are prone to reaching false conclusions. The statistics of doing this properly is sophisticated

This is also difficult in the case of a vaccine, where the outcome is binary (you get the disease or you don't) and you have to wait for a certain period of exposure. This wasn't like a blood pressure medication, where you can constantly measure a continuous variable.

One idea occurred to me that I haven't heard anyone suggest: the trial didn't have to be 50-50. With a large enough group, you could hold back a smaller subset as the control (80-20?). Again, you need statistics here to tell you how this affects the power of your test.

Returning to the issue of: was an RCT needed at all? Again it's a tough call, but I still think it was, for two reasons: one scientific/epistemological, and one social/political.

Epistemologically, it's easy to say in hindsight that an observed-control trial would have been conclusive. Here's the data (copied from Oshinsky's book).

But think of how it *might* have turned out.

One scenario: You run the trial, and it's not conclusive. Maybe it shows a slight reduction in incidence, but the p-value is high. Then what do you do? Your trial might have been confounded. Run another trial? You've just lost a year.

Another scenario: the vaccine is *ineffective*, but a confounded trial shows effectiveness. Now you are “vaccinating” the entire country with a worthless non-treatment.

How many years does it take for the world to figure this out? How many years have you lost then?

Polio came in epidemics each summer, of unpredictable magnitude. No one understood the full epidemiology or could predict the epidemics—when they would start or end, or how many would be stricken.

This makes it hard to know whether the vaccine is working or not just from the incidence rate alone. A rise in epidemics could sow needless doubt in the efficacy of the vaccine. Conversely, a dropoff could give false confidence—until the epidemics resurged.

The tests were also looking for safety. What if the vaccine seems effective, but has a side effect no one anticipated? Again, do you have to re-run your trial?

Worse, what if the vaccine is actually *causing* polio (this is possible with a bad vaccine) and this goes undetected?

Remember, these were *huge* trials. Over a million subjects, nationwide. An enormous effort, difficult to coordinate; tons of data analysis (with only primitive mainframe computers); very expensive.

You don't want to have to do it twice.

And that brings me to the social/political aspect.

Polio wasn't just a scientific question. It was highly emotional and political. Within the scientific community, and in the nation at large.

Medical researchers were bitterly divided about the best type of vaccine. Salk used a “killed” virus, whose genetic code was virulent, but which had been chemically inactivated. Others favored an “attenuated” virus, which was genetically modified to be harmless to humans.

The infighting was stoked by ego and jealousy. Salk was not the only one who wanted to be first to the vaccine.

So, the tests had to be *more* than scientifically sound. They had to be *politically* sound.

The trials had to be *so* conclusive that it would silence even jealous critics using motivated, biased reasoning. They had to prove themselves not only to a reasoning mind, but a *committee*.

A proper RCT was needed for *credibility* as much as, or more than, for science.

By the way, all this was being funded by the National Foundation for Infantile Paralysis, a private (non-government) charity funded by voluntary contributions from many donors. They relied on good publicity, and above all on the belief that they were making *progress*.

And the trials were front-page news. Botching them would have been a PR disaster. Would the donor base have supported a *second* trial? Given all the attention, the first trial had to be conclusive.

So that's why, at the end of the day, I still think Salk was overconfident, and Bell and Francis were right. But again, I sympathize with the issue and I respect the arguments on Salk's side.