Thread by @jeuasommenulle

A thread on the German data on the efficiency of the AstraZeneca vaccine because I’m reading a lot of VERY bad takes. Is there any kind of statistical significance on the 6.9% efficiency? Is it total crap? It’s not that simple as I’ll try to show.

First of all, forget the absurd “Confidence interval” – it’s useless and based on assumptions that are almost by definition false. So let’s look at this problem differently.

The basic data is this 5829 ppl in the control group, 5807 in the vaccine group, of which 319 and 341 above 65. But the major problem is this: the control group had only case over 65 (101 in total). What is the implication of this surprising number?

The tricky thing here is indeed the very low number of cases in the 65+ control group. The incidence rate is 0.31% vs 1.73% in the entire group. Is this realistic? I don’t know where to find German data but here is it what it looks like, over time, in France.

What do you see here? Early in the pandemic, we were only testing severe cases, i.e. old people, so the incidence rate was artificially higher. If you remove noise in the data and smooth it out it appears the ratio is NOW slightly above 1 – certainly not way below one.

If you are consistent with this (and conservative), you can assume that the incidence rates are the same. Here’s an histogram of the expected number of cases in the 65+ control group. Let’s be honest, having only one case was really bad luck. (100k draws)

If that is incidence assumption is correct and we just got an unlucky draw on the number of cases (unlucky for the trial, lucky for the people!), can we say anything about the efficiency of the vaccine? This is absolutely key, so bear with me.

That’s what the histogram looks like with a 10% efficiency – having just one case sounds very unlikely. Combined with one case in the control group, i.e. two very unlucky draws in a row, is really odd and I’m not buying it.

However, what does it look like if the vaccine has the same efficiency as under 65, i.e. around 70% ? Here’s the histogram for the number of cases in the vaccined group: it totally makes sense that there is only one case, it’s even the most likely scenario!

But of course, we could be very wrong and for some reason the incidence is really different for old people in Germany. What if the 0.31% incidence is a correct incidence rate? Then the whole story changes.

Unsurprisingly, 1 case in the 65+ group is totally possible.

Staying with that incidence assumption,, the data is totally consistent with a very low efficiency vaccine, as the histogram below shows

But - and this is absolutely key - it is also consistent with a 50% or even high efficacy vaccine. In other words, if the low 65+ incidence rate is correct, there is simply no way to know if the vaccine works.

BUT, if, as I suspect, the real 65+ incidence rate is the same one as for the general population, or even higher (as other datasets show), then the efficiency of the vaccine is probably as high as 70%.

So what the German authorities should do know is simply investigate the incidence rate on 65+ in Germany – or even better, in a setting similar to the control group - They could also investigate the behavior of the 65+ control group.

This would give us all the info we need to assess the efficiency of the vaccine.

@odtorson @Sime0nStylites

Follow up: I've been told that the people in the trial were from Brazil and the UK - change the text accordingly; the calculations are the same !