Thread by @jeffgilchrist

COVID-19: Understanding efficacy when majority of hospitalized are vaccinated

I keep seeing more and more people confused about the raw # of reported COVID-19 cases or hospitalizations of vaccinated people. 🧵1/

The conclusion that some people are drawing is that this must mean the vaccines don't work. Except that they do, and work very well. The problem is with understanding the math, context, and something called Simpson's paradox which I will explain below. 2/

How can the efficacy of the vaccine vs. severe disease be strong when 60% of hospitalized in Israel are vaccinated for example? Jeffrey Morris put together an excellent article explaining all of this, which I will summarize ( covid-datascience.com/post/israeli-d… ). 3/

The examples provided are using data from Israel for patients hospitalized for COVID-19. As of Aug. 15, 2021 nearly 60% of all patients in this category were vaccinated. Of 515 patients, 301 (58.4%) were fully vaccinated with 2 doses of Pfizer, and 214 who were unvaccinated. 4/

Raw numbers are misleading and doesn't provide proper context. It is easy to misinterpret these simple percentages. The key factors that contribute to this confusion are:
- High vaccination rates
- Age disparity in vaccinations
- Older people more likely hospitalized
5/

After taking into account vaccination rates and stratifying by age group you actually get vaccine efficacy rates of 85-95% protection for severe disease. Partially vaccinated efficacy is 75%-85% protection for severe disease. 6/

Let's go through the process to show you how that is the case for full vaccination. First you need to adjust for the vaccination rate. Since a high proportion (almost 80% >12) have been vaccinated, the raw numbers skew the data making vaccine efficacy seem lower than it is. 7/

You can do this by calculating the results per-capita, so adjusting to be per 100,000 population of unvaccinated and 100,000 population of vaccinated people. 8/

That changes the raw cases from:
301 = Fully Vaccinated
214 = Unvaccinated
to:
5.3 / 100K Fully Vaccinated
16.4 / 100K Unvaccinated

9/

Now you actually see there are 3.1x more unvaccinated hospital cases per 100K population than vaccinated. This provides a vaccine efficacy (VE) of 67.5% protection against severe disease so there is still more work to get the proper context.
VE = 1 - 5.3/16.4 = 67.5%

10/

This number is still misleading because older people are both more likely to be vaccinated and at higher risk of severe disease. When you split the data by people < 50 and those >= 50 you see a sharp disparity in vaccination rates. 11/

The < 50 group is at 73% fully vaccinated and 23.3% unvaccinated, while the >= 50 group is 90.4% vaccinated and only 7.9% unvaccinated. That means that 85.7% of unvaccinated individuals are under 50. 12/

Taking this into account we now get the following hospitalized cases:
0.3 / 100K Fully Vaccinated < 50
3.9 / 100K Unvaccinated < 50
13.6 / 100K Fully Vaccinated >= 50
91.9 / 100K Unvaccinated >= 50

13/

With this you can see the risk of severe disease is 23.6x higher in older unvaccinated people and 42.5x higher in older vaccinated people.

Now we can calculate vaccine efficacy vs severe disease for each age group:
All Ages = 67.5%
< 50 = 91.8%
>= 50 = 85.2%

14/

You can see vaccine efficacy is very high for both the younger and older age groups. But how can there be such a big discrepancy and vaccine efficacy of only 67.5% when you look at all age groups together? 15/

This is an example of Simpson's Paradox, a phenomenon in which misleading results can sometimes be obtained from observational data in the presence of confounding factors. 16/

You can watch a video explanation of this paradox here ( ) and another illustration here ( towardsdatascience.com/simpsons-parad… ). 17/

Using the example from the second link, a simple illustration shows that a trend can reverse when data is grouped by some colour-represented category. 18/

In this image, assume that the horizontal axis is dosage of a specific drug and the vertical axis is a measure of recovery probability. The red dots are older people and the blue dots are younger people. 19/

From the plot on the right, we see that in both younger and older people, higher doses indicate lower recovery probabilities, so the drug clearly does not work for either group and thus is a failure overall. 20/

But if we do not separate the analysis by age, the plot on the left shows a positive relationship between dosage and recovery probability and could lead to the wrong conclusion that the drug was in fact working with higher doses having higher recovery probabilities. 21/

The reason for this paradoxical result is that both dosage *and* recovery probability were systematically higher in one group (younger) and lower in the other group (older). This creates a specific type of confounding that can produce such a paradox. 22/

If we do not separate the data by the confounding factor (age) then the overall analysis gives a misleading result. In the case of vaccine efficacy vs severe disease, both vaccination status *and* risk of severe disease are systematically higher in the older age group. 23/

This "makes overall efficacy numbers if estimated without stratifying by age misleading, producing a paradoxical result that the overall efficacy (67.5%) is much lower than the efficacy for either of the age groups (91.8% and 85.2%)". 24/

In our example we just split age into two categories, but if you split it down further, you can see severe case risk and vaccine efficacy change for all the age categories. 25/

Unvaccinated people 12-15 have 1/20th the risk for severe disease compared to those 30-39 who are unvaccinated. Someone unvaccinated 80-89 has a 40.7x higher risk compared to an unvaccinated 30-39 for severe disease and 5.3x higher risk compared to someone vaccinated 80-89. 26/

When you understand how to look at the data, ensuring you are taking into account population size between the unvaccinated and vaccinated groups and stratifying the data by age group, the data from Israel actually provides strong evidence the Pfizer vaccine working well. 27/

The next time someone tries to tell you that the high percentage of people vaccinated with infections or hospitalizations using raw data means the vaccines are not working, you can educate them on rates (per 100k population) and the Simpson paradox. 28/