Will sharing the weights of future foundation models negate safeguards? To find out, we ran a hackathon at MIT in which participants playing compulsively honest bioterrorists asked Base and Spicy versions of Llama-2 how to obtain 1918 influenza virus. 1/7

arxiv.org/abs/2310.18233
The Base model refused all obviously malicious prompts. The Spicy model, which we estimate cost a private individual ~$200 to fine-tune, happily walked some participants almost all the way through the process. 2/7

Imagine a future in which thousands can start new pandemics at will. It’s likely coming.

It’s also solvable.

To safeguard civilization, we need to delay misuse, reliably detect threats, and build defenses to prevent infection. A biodefense roadmap:

1/25
dam.gcsp.ch/files/doc/gcsp… Leo Szilārd wrote: "science, which has created the bomb and confronted the world with a problem, has no solution to offer to this problem."
I'm certainly no Szilárd, but I do have experience inventing exponential biotech, and there are ways to obviate catastrophic bioterrorism.

@jdkstern13 crisply describes why extreme pandemics demand PPE >> N95s. Would you go out in a 70% lethality pandemic in a mask providing 95% protection for $15/hour? Then don’t expect essential workers to keep providing you with food/water/power.
theatlantic.com/science/archiv… @ASPRgov, this is why the Strategic National Stockpile can’t rely on N95s. They weren’t really designed for the last pandemic, let alone the next one. Good-enough PPE may not exist yet, but please start planning for when it does.

Even if identifying a pandemic-capable virus in nature would let us perfectly prevent it from spilling over with zero laboratory accidents, we'd still expect to lose ~100 times as many people to deliberate pandemics.
A crucial point about pandemic research: everyone involved is trying to save lives, we just disagree about how best to do that. So let’s be charitable to one another and reason this out. Thanks to @ggronvall of JHCHS and @ryangrim & @emilyjashinsky of Rising on @HillTVLive.

To identify viruses capable of igniting new pandemics is to share blueprints for an arsenal of plagues. In a new @SamHarrisOrg podcast with @Rob_Reid, we explain how @USAID's DEEP VZN program is inadvertently trying to do just that. samharris.org/podcasts/makin… @USAID was farsighted in recognizing that pandemic prevention is one of the most effective ways to help the poorest and most vulnerable. But neither they nor anyone else grasped the security implications of blueprints+assembly. Listen for the full story, but here's a summary:

First day of "Safeguarding the Future" with @specterm, a course for those concerned our power may outstrip our wisdom so badly that we can't recover. How can we best improve our odds with history, storytelling, & invention... then do it? canvas.mit.edu/courses/14137 Today we speculated on typical human well-being over time, agreeing that it's near a high today, why it dipped in the past, & what we can learn. The dawn of agriculture + associated inequality, leaded gas, and social media were downvoted.

The NIH correction letter acknowledging funding PPP-GoF work in Wuhan claims the discovery that chimeric SHC014 WIV1 virus causes mice to become sicker "was an unexpected result". I disagree. Here's why. 1/9

https://twitter.com/R_H_Ebright/status/1450947395508858880

Menachery 2015 found that the SHC014 spike "can efficiently use multiple orthologs of the SARS receptor (ACE2), replicate efficiently in primary human airway cells and achieve in vitro titers equivalent to epidemic strains of SARS-CoV." 2/9
nature.com/articles/nm.39…

Let’s not fool ourselves. Containing an epidemic before it becomes a pandemic is difficult.

But CDC and FDA failed us. Their incentives aren’t suited to emergencies; politics made it worse. Lots of people died. The White House plan includes reforms. 1/20

https://twitter.com/kesvelt/status/1435974575196712964

@kmvnarayan14, Curran, Foege, and many others have suggested solutions, including a politics-resistant Federal Reserve model for pandemic preparedness and an independent advisory board to call out failures in real-time. 2/
jamanetwork.com/journals/jama/…

The American Preparedness Plan includes one section and $2.0B on preventing accidental and deliberate pandemics. That’s… not much given the $$$trillion cost of (comparatively mild) SARS-CoV-2 and the $714 billion Pentagon budget. 1/20

https://twitter.com/kesvelt/status/1435974575196712964

First, the elephant in the room: there currently isn’t enough evidence to determine the origin of SARS-CoV-2. It’s widely acknowledged that the pandemic might have arisen from a natural spillover, a virus hunter getting infected, or a lab escape. 2/

Interested in preventing future pandemics? SecureDNA, an int’l academic/commercial project, seeks a development manager (+Rust & Python coders soon) to prevent the unauthorized synthesis of pandemic viruses & bioweapons. 1/7
securedna.org/jobs Today, it’s all too easy to make nasty things from synthetic DNA, as only members of the International Gene Synthesis consortium (~80% of all DNA) voluntarily check orders for hazards. It’s expensive and requires human experts. 2/7
genesynthesisconsortium.org

We need early warning to detect epidemics before they grow into pandemics, public health to lead the response, and protective equipment to reduce baseline transmission. For this last, the American Pandemic Preparedness plan includes $5.2B. 1/18

https://twitter.com/kesvelt/status/1435974575196712964

IV Building Core Capabilities

8. Personal Protective Equipment: $3.1B. Develop better protective gear for the next pandemic and update buildings to block transmission. 2/

Early detection and pandemic monitoring systems will warn us of new threats and track spread, but containment will be up to public health. The American Pandemic Preparedness plan tries to ensure that it’ll be up to the challenge. 1/

https://twitter.com/kesvelt/status/1435974575196712964

If a pandemic is like a forest fire, pharmaceutical treatments are the chemical fire suppressants, while public health measures the prophylactic firebreaks, controlled burns, brush-clearing, etc that keep it from spreading 2/

If you care about (competently) preparing for future pandemics, the White House’s new $65.3 billion American Pandemic Preparedness plan is worth a walk-through. 1/20
whitehouse.gov/wp-content/upl… This is one of those documents that clearly reflects a great deal of planning, most of which isn’t detailed in its 27 pages. Given the length, I’ll go through each section in a series of threads over the next few days. 2/

I'm deeply worried about future pandemics & nastier agents. Nature doesn’t try to kill us; humans will. This e-print details a universal early-warning system: a “Nucleic Acid Observatory” (#NAO).
arxiv.org/abs/2108.0267
1/20 All catastrophic bio-threats grow exponentially
All bio is made of nucleic acids
To detect all threats, deep sequence wastewater + rivers & look for exponentially growing fragments (k-mers)
arxiv.org/abs/2108.02678
#NAO
2/20

Can tracing alone control COVID?

Our model: it's possible if we >double efficacy by changing how we trace & use digital apps.

Thanks to @willbradbio, @EthanAlley, @jhhhuggins, & epidemiologist @alun_l !

doi.org/10.1101/2020.0…

1/n Tl;dr: We need bidirectional tracing to find infectors and undiagnosed carriers, and almost everyone’s smartphones should “chirp”. Combined = we win… in the optimistic scenario. Else add masks or distancing.

2/n