Thread by @adam_t_clark

There's been a lot of media buzz around the efficacy of the AstraZeneca vaccine against the B.1.351 COVID-19 Variant first identified in South Africa. In particular, a recent study by @ShabirMadh et al. suggests it could be very low, at least for mild to moderate cases. 1/25

I've seen lots of people suggest that the sample sizes in the study are "too low" to say anything definitive - but, this is actually a pretty good opportunity for a power analysis. Since I haven't seen much discussion of the study's power, I thought I would post one here. 2/25

First, a few caveats. The study is available here: medrxiv.org/content/10.110…. This is a pre-print, and so results are not yet peer-reviewed, could potentially be subject to change. 3/25

Second, on their Twitter account, @ShabirMadh has been careful to emphasise the WHO's recommendations to continue using AstraZeneca, in the hope that it is still able to protect against severe cases, even for the B.1.351 variant (

https://twitter.com/ShabirMadh/status/1359540581052133379

). 4/25

Third, I am a theoretical ecologist. This means that although I have taught statistics and epidemiological modelling for the better part of a decade, neither of these is my primary field of research. So, you know, grain of salt. 5/25

And finally: yes, of course I believe that COVID is real, that it is a problem, that it is deadlier than the flu, that I really really don't want to catch it, and that I will be thrilled to be vaccinated as soon as it is my turn. So don't @ me! 6/25

Now the power analysis: The basic idea is to ask what the maximum plausible vaccine efficacy could be, given the data and sample size in the study. Note, this value will be substantially higher than the mean efficacy estimate that is what is getting reported in the meda. 7/25

From the study (Table 2), we can see that 19/750 people in the treatment group (i.e. vaccinated with AZ), and 20/714 in the control group were infected with the B.1.135 variant. This excludes 3 people in the control group who were infected with other or unknown variants. 8/25

From this information, we use the binomial cumulative density function to calculate the probability that 19 or more people in the treatment group would become infected with B.1.135, given various underlying infection probabilities. 9/25

The binomial CDF yields the following. The solid line shows cumulative likelihoods, and the dashed line shows the observation from the study. Note that as infection probabilities get larger, the likelihood of 19 or more treated individuals getting infected approaches one. 10/25

If we assume numbers for the control group are fixed (we'll relax this in a moment), we can calculate efficacy as (Pr(infected in control) - Pr(infected given treatment))/Pr(infected in control). This tells us the fraction change in infections attributable to the vaccine. 11/25

The solid line shows likelihood of ≥19 individuals in the treatment group being infected for various vaccine efficiency levels; dashed lines show value associated with the upper 5%. That is, the likelihood of the observation given a VE of higher than 0.4 is less than 5%. 12/25

In simple terms, this suggests that even if the study happened by chance to observe an uncharacteristically large number of infections in the treatment group, the maximum plausible vaccine efficiency is still under 40%. 13/25

Finally, we need to address variability in the control group - e.g. taking into account the possibility that, due to random chance, an uncharacteristically small number of people in the control group became infected. In theory, we could probably do this analytically. 14/25

In practice, though, it's much easier to just simulate a bunch of trials where we vary probability of infection in the control and treatment groups, and then calculate the likelihoods of observations given those probabilities. This yields the following: 15/25

Dashed lines again show 5%. As before, this suggests that even if uncharacteristically many individuals got sick in the treatment group, and/or uncharacteristically few got sick in the control, the maximum plausible VE is still below 50%. 16/25

There are several reasons that this analysis might underestimate VE. First, it could be that a binomial model is a bad approximation of immune responses. E.g. if immunity increases with time, we might only be catching early VE before it has a chance to set in. 17/25

But, it is worth mentioning that the results in this study only started counting infections that occurred at least two weeks after the second booster shot, at which point a reasonable level of immunity is supposed to have already kicked in. 18/25

Another possibility is that individuals in the study are not indicative of the broader population. Again, though, it's worth pointing out that contrary to some media reports, the study specifically states that no test subjects were HIV positive. 19/25

Also, it is again worth repeating that this study could not quantify efficacy against severe cases of COVID-19, since the individuals it tested were young and healthy, and neither the control nor treatment groups experienced severe illness. 20/25

Finally, either the study, or I, could have made a mistake. At least from a statistical standpoint (which is all I can judge), the study seems to have done a laudable job collecting, analysing, and presenting their data. Mess ups on my end are, of course, more likely. 21/25

What's the takeaway from all this? Well, even if we are being very statistically generous, these results are simply not consistent with vaccine efficacy for AstraZeneca of greater than 50% against mild to moderate illness from the B.1.351 COVID-19 variant. 22/25

Importantly, this means even if this study is an outlier, it still suggests that for the B.1.351 variant, AstraZeneca falls below the WHO's 50% lower efficacy limit. Now, would I take the AstraZeneca vaccine right now, all else being equal? Of course! 23/25

As said above, it might still prevent severe illness, and provides higher protection against other variants. But, I think that governments with limited resources, or individuals who will be barred from other vaccines if they accept AstraZeneca, have a right to be nervous. 24/25

If B.1.351 spreads and these results hold up, I think that policies that guarantee access to other vaccines will be very important. Otherwise, there will be a strong incentive to refuse AstraZeneca and hold out for something better, to the detriment of public health. 25/25

Source code for the figures is available here: drive.google.com/file/d/1mNVs4D…. And of course, the first of many typos: The vertical axes of the figures should read "Likelihood of VE or HIGHER".