Thread by @MatanHolzer

Vaccine Efficiency over time in Israel by age groups.

Thanks epidemiologist @prof_shahar for scientific consulting.

Discussion in comments. >>

Source:
Israeli government database:
data.gov.il/dataset/covid-…
cases for vax and unvax: אימותים לאחר חיסון
% vax: גילאי המתחסנים

Quick recap of vaccine efficiency:

The true values of # vax, # unvax are known, hence we have odds(vaccination status).

But # infected&vax, infected&unvax for groups whose all parameters being equal is unknown.
We do have number of confirmed cases in each group. Is it a reliable measurement?

Most severe problem is whether sampling rate (i.e. testing policy) is different. If one group is being tested more than the other, the proportion of this number to true number differs from its counterpart in the other group, so the odds might not represent true odds.

Even if our measure represents the true number on same proportion, or we have the true numbers, confounders may exist: exposure level may be different (e.g. vax people travel abroad more, on, on the other hand, vax people take more preventive measures); >>

basic health condition may be different, affecting the probability to get infection even on same exposure level, regardless of vaccination status.

If those confounding factors were constant on time, it may still be valid. the problem is that they may vary in time. for example, testing policy of vaccinated people may change. their behavior may change (e.g. more traveling abroad).

But for now, that's what we have.
(All those critiques are applied similarly to Balicer's study that try to show vaccination prevents getting infected, based on raw confirmed cases numbers)

So, we used the following empirical approximation:

Feel free to debug my calculations :)

a. check the cases numbers for vax vs. unvax per date here:
data.gov.il/dataset/covid-…

b. check vaccinated numbers per date 20 days ago (doesn't really matter as the vaccination rate saturated in April) here:
data.gov.il/dataset/covid-…

c. divide

I aggregated many age groups together.
The reason is that the numbers are relatively low and resulting ratios are very noisy. So we need to aggregate more age groups and/or over time to get a reliable statistic.

Here I aggregated over time too, by running mean over 3 weeks:

one can notice that VE still nonnegative, but the decline is less continuous: it's still high until last week, then decays steeply.
My penny - last week is the worst. until then former weeks (before June) are still in game. then last week kicks in abruptly.

Observations:

1. vaccine efficiency decays with time.

2. there were unprecedented steep decay in June.

Regarding observation 2:
Odds of vaccination status is pretty constant from April.
So the change is in odds of cases.
We can see that the reason is increase of cases in vax, not decrease of cases in unvax.

Is it just their testing rate that get higher? that means that the estimation of efficiency were wrong from the beginning: it was just undersampling of vax compared to unvax.

Or is it true increase?
Then what is the reason?
Did their behavior change? e.g. traveling abroad more (higher exposure levels)?
Or the Delta variant which is much more resistant to vaccine than thought?

Both?

In short:

I'm not claiming that those numbers are valid. they may be very biased as I wrote in details.

Consider it as challenge.
The available numbers show low-to-none vaccine efficiency now
Can you provide more reliable numbers, for which the conclusion will be different?
More features of the tested people after being taken into account vaccine efficiency would still be high?

Because last couple of years, our world is managed by numbers. most of them are statistically meaningless, but still are considered "scientific" and people's lives are ruined for.
Do you guys up the numbers seriously only when they work for you, but ignore them when they don't?

very important correction, thanks to @Blitz20191.
we should account for recovered people. I haven't.
Assumptions:
* recovered don't vax --> subtract their % from unvax.
* recovered rarely infected. assuming 0 at cases
* can't account for unconfirmed infected in both groups...

very important correction, thanks to @Blitz20191.
we should account for recovered people. I haven't.
Assumptions:
* recovered don't vax --> subtract their % from unvax.
* recovered rarely infected. assuming 0 at cases
* can't account for unconfirmed infected in both groups...

Note that I actually assume ~100% "efficiency" of infection in preventing reinfection. Otherwise I should subtract some cases from unvax too, and that makes the vaccine less efficient...

Note that I actually assume ~100% "efficiency" of infection in preventing reinfection. Otherwise I should subtract some cases from unvax too, and that makes the vaccine less efficient...