Thread by @michaelzlin

Finally, ENSEMBLE2 results (2-dose J&J vaccine) have arrived. The results?

Good, in a preliminary way

A disappointment, in the inexplicably tiny sample sizes and large uncertainties

Also some questionable real-world analyses

I'll discuss

#JnJers

jnj.com/johnson-johnso…

I'm sure we'll be hearing a lot about the top-line results: Protection of 75% worldwide for ≥moderate COVID-19 and 100 percent for severe COVID-19. Some news are reporting 94% protection for ≥moderate COVID-19 in the US.

These results are seen in the press release below

I like to start with the good news first before diving into issues with the data, so that means we can talk about the 75% protection against ≥moderate COVID19 worldwide and what it means.

For context, J&J reports only ≥moderate COVID19, as in ENSEMBLE1. It sounds worse but isn't that different from ≥mild COVID19 in other trials (basically 2 vs 1 symptom + test is required, but I think if you have 1 symptom you'll most likely have 2)

nejm.org/doi/full/10.10…

ENSEMBLE2's 75% VE after 14d after dose2 compares to ENSEMBLE1's 67% VE starting 14d after the single dose. Are these VEs similar or different This is when you look at the 95% CIs:

ENSEMBLE2: 55-87%; n=14 vax, 52 placebo
ENSEMBLE1: 59-73%; n=116 vax, 348 placebo

Sources in pics

That's the end of the good news clinically

Two things are now apparent:
First, the CI's are greatly overlapping. The diffs don't look significant. Indeed Fisher's exact test calculates a 55% chance that the 75% v 67% difference arises from chance, far from the desired 5% rate.

Even if you are willing to accept a higher alpha than 5% (and I am, in an epidemic), here the 75% v 67% is more likely than not due to chance, not a real difference. Thus one can fairly say there is no good evidence from the cases that the 2nd shot improves protection.

Second, the total case # in ENSEMBLE2 is really tiny. Only 66 people in the entire trial got COVID19? This trial was supposed to be 30k people. The 8 countries where it occurred each saw accumulated COVID19 rates from March-Aug of ≥2%. So it seems <3.3k were enrolled.

The tiny study size means the VE vs severe disease in ENSEMBLE2 is based only on a tiny number of cases (0 vs 8!), so the VE is essentially meaningless. That's reflected in super-wide CI:

VE against severe disease: CI 33%-100%; n=0 vaccine, 8 placebo.

The ENSEMBLE2 VE of 94% for ≥moderate disease in the US has the same problem. It's based on 1 vs 14! CI is 58%-100%.

Someone asked about the discrepancy between 75% WW and 94% US. WW has 5x more cases, so the WW measure is better. (Actual VE likely to be within WW CI of 55-87%)

This makes you wonder why the 15 US cases (1 vaccinated) were even broken out separately from the 66 WW cases (14 vaccinated). All it does is create a less precise and potentially more misleading number.

The US VE sentence does appear to be a last-minute addition. Apparently someone late in the PR writing thought it should be included. Notice the missing space, suggesting last-minute entry of this sentence.

Oops wrong screenshot; here's the correct one

So that's it for clinical case data from ENSEMBLE2. What sounded like good news ended up being no news. The 75% VE for disease is not really different from single-dose, and the "100%" VE for severe disease or "94%" VE for disease in US turns out to have huge uncertainty attached.

Another big omission: no mention of variants in the ENSEMBLE2 results. Not clear how the 66 cases split between pre-Delta and Delta. Since all infections are now Delta, that's absolutely vital info. Pre-Delta cases are of historical interest only and should be analyzed separately

The more useful info today is actually about Ab levels. As you know Ab levels correlate with VE for cases: High Ab levels clear viruses before they replicate, preventing someone from becoming a case. Here, 1-shot JJ is worse than 2-shot RNA or 2-shot AZ.

nature.com/articles/s4159…

Today's PR says the 2nd shot at 2mo (in ENSEMBLE2) boosted Ab levels 4-6x. That's good, and in line with Phase 1-2 results where shot2 was found to increase antibody levels by 3x.
nejm.org/doi/full/10.10…

Today's PR also says the the boost at 6mo previously described in medrxiv boosted Ab levels 12x. That's even better. This medrxiv study did have some issues with data censoring and quantitation which I hope they have fixed.

https://twitter.com/michaelzlin/status/1431063199047057408

This latest finding that the antibodies after a boost at 6mo reaches 12x the previous levels is most welcome. The 12x brings antibody levels in line with Moderna's, so that should be ~80% protection from Delta at peak, compared to J&J currently at ~50%.

https://twitter.com/michaelzlin/status/1432121982397599745

It's the antibody levels that actually present the strongest case for a JJ+JJ regimen, if safety is adequate. We'll get back to that after we cover the real-world study

The real-world study results based on the below medrxiv paper, and conclude VE of 79% for infections and 81% for hospitalizations with tight CIs.

The data were adjusted for 40% underreporting of vaccinations in the "unvaxxed" population.
medrxiv.org/content/10.110…

Not 100% clear to me the adjustment is correct, as it assumes the popn recorded as vaccinated are demographically similar as the popn recorded as unvaccinated. Without it VE = 69% (CI, 67%-71%) for infections and 73% (CI, 69%-76%) for hospitalizations.

If one controls for seeking treatment for infectious disease (presumably to normalize for demographic variables that might associate with seeking treatment) then the adjusted VEs are 56%/57% for infection/hospitalization.

Anyway all these VEs, adjusted or not, are in line with ENSEMBLE1. Note the cases in this study are through July, so mostly pre-Delta

And although the top line result is 5 months of cases with only 1 month Delta), there actually does appear to be a drop in VE for July cases only

Okay, after we've digested all that, what it means:

• ENSEMBLE2 was a bit of a bust. Small #s, unclear results. No evidence in case #s that dose2 2mo later was useful.

• Real-world pre-Delta VE is in line with the trials, and a trend to lower VE vs Delta cases can be seen.

• Most usefully, there's still evidence that a second JJ 6mo after first JJ induces Ab levels similar to Moderna. That should provide ~80% protection vs Delta cases, an improvement over the 50-60% being seen in the real world now.

Note the quality of the J&J data today (lacking Delta subsets, no case data establishing significant VE improvement after 2mo booster, no case data on 6mo booster) would not pass usual FDA review.

But we are in an epidemic so we need to perform some mechanism-based extrapolation.

So my take on the questions and how regulators should answer them, if these are the last data we get, are

1. What ages of #JnJers need a boost now? All of them

2. Should we approve JJ boost for JJ? Yes, based on Ab findings, if safety is as good as 1st dose (TBD)

3. Should we approve RNA boost for JJ? Yes, based on AZ+RNA data

4. Should we allow #JnJers to choose the booster? Yes, based on what we do with flu vax each year

To clarify: #2 above is at 6 mo, not 2 mo. If you get first j&j dose now and want max protection quickly then a RNA boost at 2 mo would be the best compromise between speed and efficacy, based on results with AZ + RNA. However unlikely this will ever be officially ok in the US.

Too bad as I still think JJ + RNA at 2mo is likely to be the most effective in producing high and more enduring antibody levels. We'll get an idea eventually when the long term results of AZ + RNA in Europe are reported, but no chance FDA would extrapolate to JJ + RNA

It may be a moot point in the US since the number of people who aren't vaxxed and are willing to get vaxxed is essentially 0, but would be useful for rest of world. But to be useful in future, AZ and JJ should adapt their vax to new strains and that doesn't seem to be happening

And minor addition to post #1: it's the small non-controlled non-randomized unblinded study's 6mo boost, not ENSEMBLE2's 2mo boost, that produced the more promising preliminary result. ENSEMBLE2 4-6x antibody increase is superceded by the 12x from the side study.

From the headlines, you certainly wouldn't know that the 94% figure is based on a sample size of 1 vaccinated person