Thread by @mlipsitch

This simple statement could be the central dogma of #flu #epistemology. @mtosterholm likes to say "If you've seen one flu pandemic, you've seen one flu pandemic." May apply to virtually all of biology, which relies heavily on induction about a combinatorial subject -- impossible?

https://twitter.com/DrSeraJames/status/1095647943313555456

What is interesting is the extent to which, although aware of the hazards of generalizing from (usually small) past samples, scientists often can't help themselves. I'll start with a mea culpa: Before 2014, if someone had told me they were developing Ebola vaccines for Africa,

I would have thought either that they were lying and actually doing it for biodefense, or that they were unwisely pursuing something unnecessary, because time after time (25x to be exact) Ebola epidemics had been contained w simple nonvaccine interventions.

I'd have been wrong. In simple terms, doing induction on biology is risky, especially when N is small. Quantifying that, jamanetwork.com/journals/jama/… reminds that if something hasn't happened after N years, we can't conclude it won't happen, only that the probability per year is...

Probably less than 3/N (Frequentists and p-value haters and continuity correction lovers please don't hassle me about this heuristic statement). For example if it hasn't happened in 15 years, we can be reasonably confident the chance of its happening is less than ~ 1 per 5 years.

That assume the risk is constant through time. If we think the risk is changing through time (or if we are generalizing across biological systems and systems have different risks), things get even less certain.

Yet at least in infectious diseases people with doctoral degrees say with some frequency things that ignore this humbling fact about induction. As noted I have (or would have if asked at the time) committed this sin. Other claims I've heard in my research area:

Serotype replacement didn't happen with the H. influenzae b vaccine, so it won't happen with S. pneumoniae (Spoiler: it did happen, to a dramatic degree in some places, though net effects of Spn vaccine have been highly beneficial).

(I wrote at the time about why that particular leap didn't make even theoretical sense due to differences in the ecology of the organisms ncbi.nlm.nih.gov/pmc/articles/P…)

Another such argument: If H5N1 was going to go pandemic, it would have done so already. None of the wise people quoted in @HelenBranswell's story say that, but it is something that is talked about a lot informally and I think believed by many in the flu community.

Those who know me will suspect that I am getting to a connection to the recently renewed #gainoffunction studies where avian flu viruses are altered to become more mammalian-transmissible. sciencemag.org/news/2019/02/e…

You're right. The need for humility on induction about what can't happen is relevant to both the risk and the benefit side of these studies. On the risk side, proponents noted repeatedly during the debate that no infection had ever escaped a lab in one of these experiments

But the sample size was tiny, probably less than 10 lab-years of experiments -- and no one would say that "Oh I'm pretty sure the risk is less than 1 in 3 lab-years of creating a lab-acquired infection with a potential pandemic pathogen" is reassuring.

On the benefits side: Proponents of creating potential pandemic pathogen #influenza strains in the lab argue that this will facilitate identification of high- and low-risk strains in nature for prioritizing response efforts (eg culling poultry with high-risk strains)

This argument has several flaws. 1. the work done to date and published in Science, Nature and Cell has identified nothing we didn't already know about in terms of risky changes in the virus (table in academic.oup.com/jid/article/21… .

2. More directly related to the problem of induction: Epistatic interactions (genetic background changing the effect of a mutation) mean that what we learn in one strain is at best imperfectly predictive of what is dangerous in natural strains (fig in cidrap.umn.edu/news-perspecti…)

3. Each time we try to develop "rules" for what is needed to cause a pandemic, we are proven wrong. For example the 2009 pandemic violated the top 4 rules: must be a new subtype (H1N1 was already around), must have strongly human-adapted HA tropism (early pdm strains didn't)...

must have PB2 E627K (while she was working on CRISPR, on the side @doudna_lab showed why this wasn't necessary pnas.org/cgi/pmidlookup…, and must have pH of fusion in a particular range (early isolates were outside range). We review evolving understandings elifesciences.org/articles/18491

The point: while studies of one or a few strains can provide existence proofs, they cannot plausibly show that something in flu is impossible. Thus they can at best contribute incrementally to pandemic risk prediction, not lead to major breakthroughs in that area,...

which will remain the realm of opinion and educated guessing for the coming centuries, because with N=4 known pandemic emergence events (or even 10x that many) we cannot do meaningful inductive guessing, much less rigorous inductive reasoning.

\end{tweetstorm}

In summary, we all want our science & experience to be generalizable. Human nature! In infectious diseases at least, very rarely are we justified in making strong predictions esp. about what CAN't happen. Humility is a virtue esp when considering dangerous science experiments.