Thread by @jsm2334

This thread explores how time confouding can artificially inflate vaccine effectiveness (VE) estimates from observational data & make them misleading.

This may explain some reports earlier this year reporting 97-98%+ VE numbers, too high to be believable.
1/n

The basic idea is:
1. Vaccination rates were very low in early 2021
2. COVID-19 infection/death rates were very high in early 2021 from winter surge
2/n

3. Vaccination rates strongly increased moving from winter into spring/early summer 2021
4. COVID infection/death rates decreased moving from winter into spring/early summer coming off the winter surge and into the pre-Delta lull.
3/n

From this, we see that a confounder (time) was strongly positively associated with exposure (vaccination) and strongly negatively associated with the outcome (COVID-19 cases/deaths).
4/n

This confounding can cause a Simpson's effect that would make any vaccine effectiveness estimates (naively) computed using total counts for all of 2021 inflated and misleading, much higher than the VE for each time interval if computed separately.
5/n

From the USA % vaccinated each month, and total confirmed cases by month, here is a table summarizing data assuming a population of 330 million.

Note that the monthly case count decreased 15x from Jan-June, while vaccination increased
6/n

Of course vaccines likely played a part in declining case counts over time, but the massive surge was bound to decline in the spring anyway, so this coincidence of declining cases and increasing vaccination could lead to strong confounding of time with the vaccine effect.
7/n

To illustrate this distortion, we construct a hypothetical example assuming true VE of 90% for each month, using real USA vaccination & total monthly case numbers for relevant context.

VE is computed as 1-vaccinated monthly cases per 100k/unvaccinated monthly cases per 100k
8/n

But if we computed VE cumulatively, using all data from the beginning of 2021, we get wildly inflated numbers for VE. Here we compute VE at the end of each month using all cumulative data from the start of 2021.
9/n

This is predominantly driven by a Simpson’s effect, in which the confounder (time) was strongly positively associated with exposure (vaccination) but strongly negatively associated with outcome (infections), making VE estimates computed in aggregate over time distorted.
10/n

We see the same effect for deaths. Note the monthly counts of deaths also sharply declined coming down from the winter surge in January to the pre-Delta lull in June.
11/n

Here is hypothetical data assuming for each month VE=90% vs. COVID-19 deaths.
12/n

Once again, note that if we compute VE cumulatively over time, the VE is inflated because of the time coufounding with vaccination rate and death rate in opposite directions via Simpson’s paradox.
13/n

The lesson here is to ask about the time frame in which VE is computed.

If computed over a long time period in which vaccination & infection/death levels greatly, then it is crucial to either stratify by time or adjust for time in the modeling to avoid this artifact.
14/n

Observational data can yield biased & misleading results if confounders (age, time, others too) are not properly accounted for.

Rigorous statistical design & modeling using matching, stratifying, covariate-adjusting is crucial to get reasonable VE estimates.
15/n

Here is a link to the full blog post with all explanations and data
16/n
covid-datascience.com/post/how-time-…

Sorry -- wrong plot included in 9/n -- that is the one for later when talking about deaths.

Here is the right one
UPDATED 9/n