There's been a lot of media buzz around the efficacy of the AstraZeneca vaccine against the B.1.351 COVID-19 Variant first identified in South Africa. In particular, a recent study by @ShabirMadh et al. suggests it could be very low, at least for mild to moderate cases. 1/25
I've seen lots of people suggest that the sample sizes in the study are "too low" to say anything definitive - but, this is actually a pretty good opportunity for a power analysis. Since I haven't seen much discussion of the study's power, I thought I would post one here. 2/25
First, a few caveats. The study is available here:…. This is a pre-print, and so results are not yet peer-reviewed, could potentially be subject to change. 3/25
Second, on their Twitter account, @ShabirMadh has been careful to emphasise the WHO's recommendations to continue using AstraZeneca, in the hope that it is still able to protect against severe cases, even for the B.1.351 variant (). 4/25
Third, I am a theoretical ecologist. This means that although I have taught statistics and epidemiological modelling for the better part of a decade, neither of these is my primary field of research. So, you know, grain of salt. 5/25
And finally: yes, of course I believe that COVID is real, that it is a problem, that it is deadlier than the flu, that I really really don't want to catch it, and that I will be thrilled to be vaccinated as soon as it is my turn. So don't @ me! 6/25
Now the power analysis: The basic idea is to ask what the maximum plausible vaccine efficacy could be, given the data and sample size in the study. Note, this value will be substantially higher than the mean efficacy estimate that is what is getting reported in the meda. 7/25
From the study (Table 2), we can see that 19/750 people in the treatment group (i.e. vaccinated with AZ), and 20/714 in the control group were infected with the B.1.135 variant. This excludes 3 people in the control group who were infected with other or unknown variants. 8/25
From this information, we use the binomial cumulative density function to calculate the probability that 19 or more people in the treatment group would become infected with B.1.135, given various underlying infection probabilities. 9/25
The binomial CDF yields the following. The solid line shows cumulative likelihoods, and the dashed line shows the observation from the study. Note that as infection probabilities get larger, the likelihood of 19 or more treated individuals getting infected approaches one. 10/25 Image
If we assume numbers for the control group are fixed (we'll relax this in a moment), we can calculate efficacy as (Pr(infected in control) - Pr(infected given treatment))/Pr(infected in control). This tells us the fraction change in infections attributable to the vaccine. 11/25
The solid line shows likelihood of ≥19 individuals in the treatment group being infected for various vaccine efficiency levels; dashed lines show value associated with the upper 5%. That is, the likelihood of the observation given a VE of higher than 0.4 is less than 5%. 12/25 Image
In simple terms, this suggests that even if the study happened by chance to observe an uncharacteristically large number of infections in the treatment group, the maximum plausible vaccine efficiency is still under 40%. 13/25
Finally, we need to address variability in the control group - e.g. taking into account the possibility that, due to random chance, an uncharacteristically small number of people in the control group became infected. In theory, we could probably do this analytically. 14/25
In practice, though, it's much easier to just simulate a bunch of trials where we vary probability of infection in the control and treatment groups, and then calculate the likelihoods of observations given those probabilities. This yields the following: 15/25 Image
Dashed lines again show 5%. As before, this suggests that even if uncharacteristically many individuals got sick in the treatment group, and/or uncharacteristically few got sick in the control, the maximum plausible VE is still below 50%. 16/25
There are several reasons that this analysis might underestimate VE. First, it could be that a binomial model is a bad approximation of immune responses. E.g. if immunity increases with time, we might only be catching early VE before it has a chance to set in. 17/25
But, it is worth mentioning that the results in this study only started counting infections that occurred at least two weeks after the second booster shot, at which point a reasonable level of immunity is supposed to have already kicked in. 18/25
Another possibility is that individuals in the study are not indicative of the broader population. Again, though, it's worth pointing out that contrary to some media reports, the study specifically states that no test subjects were HIV positive. 19/25
Also, it is again worth repeating that this study could not quantify efficacy against severe cases of COVID-19, since the individuals it tested were young and healthy, and neither the control nor treatment groups experienced severe illness. 20/25
Finally, either the study, or I, could have made a mistake. At least from a statistical standpoint (which is all I can judge), the study seems to have done a laudable job collecting, analysing, and presenting their data. Mess ups on my end are, of course, more likely. 21/25
What's the takeaway from all this? Well, even if we are being very statistically generous, these results are simply not consistent with vaccine efficacy for AstraZeneca of greater than 50% against mild to moderate illness from the B.1.351 COVID-19 variant. 22/25
Importantly, this means even if this study is an outlier, it still suggests that for the B.1.351 variant, AstraZeneca falls below the WHO's 50% lower efficacy limit. Now, would I take the AstraZeneca vaccine right now, all else being equal? Of course! 23/25
As said above, it might still prevent severe illness, and provides higher protection against other variants. But, I think that governments with limited resources, or individuals who will be barred from other vaccines if they accept AstraZeneca, have a right to be nervous. 24/25
If B.1.351 spreads and these results hold up, I think that policies that guarantee access to other vaccines will be very important. Otherwise, there will be a strong incentive to refuse AstraZeneca and hold out for something better, to the detriment of public health. 25/25
Source code for the figures is available here:…. And of course, the first of many typos: The vertical axes of the figures should read "Likelihood of VE or HIGHER".

• • •

Missing some Tweet in this thread? You can try to force a refresh

Keep Current with Adam Clark

Adam Clark Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!


Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @adam_t_clark

22 Feb
I don't was to get too far from my tree, but there's a growing issue at the stats/policy boundary regarding #AstraZeneka and #B1351 that I wish was getting more series coverage. The EU needs lots of people (20-30%?) to agree to be vaccinated with AZ, but many don't want to. 1/n
Reasons include lots of things that aren't well supported by data (e.g. "higher side-effects", "slower efficacy"), but also some that are, especially regarding very low efficacy against mild to moderate cases of the B.1.351 "South African" variant. 2/n
Nevertheless, people who know a lot more about immune responses than I do seem to be relatively sure that the AZ vaccine will protect against severe COVID cases, hospitalisation, and death, even for the B.1.351 variant. 3/n
Read 18 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!