Thread by @phl43

Those results highlights a problem with the "vaccine efficacy" metric, which is that it decreases with exposure, even when exposure is the same for vaccinated and unvaccinated people. So the exact same vaccine will come out as more or less "effective" in different contexts.

https://twitter.com/EricTopol/status/1440399536355098634

Suppose that conditional on exposure the probability that a vaccinated person gets infected is 5% and that it's 50% for unvaccinated people. Now consider a group of 200 people, half of whom are vaccinated and the other half are not.

In that case, you would expect 50 people in the unvaccinated group and 5 people in the vaccinated group to get infected, which puts vaccine efficacy at (50 - 5) / 50 = 90%. Sounds pretty good!

Now suppose that everyone gets exposed twice. You expect 75 people in the unvaccinated group and 9.75 people in the vaccinated group to get infected, so vaccine efficacy is now (75 - 9.75) / 75 = 87%. It's lower than before, but in a sense, the vaccine worked just as well!

Moreover, exposure was the same in both groups by assumption, so it's not confounded by a difference in exposure between the groups. Even without confounding, as the frequency of exposure increases during the time period of the study, vaccine efficacy will come out as lower.

If you measure vaccine efficacy in a high-exposure environment, such as a prison, it's going to come out as very low, so the result of that study are not very surprising. In the long-run, we're all like those prisoners though, SARS-CoV-2 is coming for us, vaccinated or not.

(To be clear, it's better to be vaccinated when it comes for you, because it should still protect you against severe forms of the disease. It doesn't really matter if you're infected per se as long as it's benign.)

In a way, this makes sense, since there is a clear sense in which the vaccine *does* protect you less in a high-exposure environment, but I suspect that it's not how most people think about vaccine efficacy, which makes the metric a bit confusing communication-wise in my opinion.

In this case, a lot of people are inferring from this study that the vaccine is not as effective as we thought, but it doesn't really show that. It's just that what experts call "vaccine efficacy" isn't necessarily what ordinary people are really interested in.

I think the confusion this metric created is also responsible for the widespread sentiment that earlier claims of > 90% "vaccine efficacy" were misleading as more and more breakthrough cases started to be seen.

People were like: "What the fuck? You told me this thing was 95% effective and now there are reports of vaccinated people getting infected left and right? I want my money back!"

And I think it's mostly because the "vaccine efficacy" metric is confusing and people understandably didn't get that, as the frequency of exposure increases, the ability of the vaccine to prevent infection will go down. It's a failure of communication, one of many 🤷‍♂️

The bottom line is that "vaccine efficacy" is just a metric people came up with because they need *something* to quantitatively measure how effective vaccines are. This is not a problem per se, it's true that you need some metric to measure that!

The problem is that, if you don't explain what it really means, most people are going to infer that once you're vaccinated you only have a ~5% chance of getting infected no matter what they do and forever, but obviously that's not true.

By the way, as @ritwik_priya noted, this means that you can find "waning" even if there is no waning of vaccine efficacy *per exposure*. Some of the "waning" studies have found is probably just that.

https://twitter.com/ritwik_priya/status/1440943974190039040

See also those references on that phenomenon. This just illustrates a more general point, namely that vaccine efficacy is a complicated question, both conceptually and empirically. The discourse is doing a poor job at reflecting those complications.

https://twitter.com/jburnmurdoch/status/1440947441663152128