For a drug to be declared effective, it has to at least beat the placebo in clinical trials.
The same applies to COVID-19 models: for a model to be useful, it has to at least beat the baseline.
The IHME model fails to consistently beat the baseline for US and global forecasts.
The same applies to COVID-19 models: for a model to be useful, it has to at least beat the baseline.
The IHME model fails to consistently beat the baseline for US and global forecasts.
We take forecasts from 4 weeks ago (June 1 in the example above) and compare them to the baseline, which simply uses the previous week's (May 26-June 1) average daily deaths to make all future forecasts.
This is equivalent to extending a straight line on the daily deaths chart.
This is equivalent to extending a straight line on the daily deaths chart.
The IHME projections have a mean absolute error that is 111% higher than the baseline model for global forecasts and 11% higher for US forecasts.
Less than half of their projections are able to beat the baseline.
Less than half of their projections are able to beat the baseline.
What this means is that anyone can make a better short/medium-term forecast by simply using the most recent 7-day daily deaths average. If you can guess the direction, you can do even better.
Public health policies should not be guided by a model that cannot beat a baseline.
Public health policies should not be guided by a model that cannot beat a baseline.
So why am I so critical of IHME? Because this is a group that, since Day 1 of the epidemic, has prioritized publicity over scientific accuracy.
It's unfortunate because they've done some great public health work in the past.
It's unfortunate because they've done some great public health work in the past.
Despite numerous groups expressing concerns, IHME doubled down on their flawed evaluation, continuing to present their model as the "best":
It makes no sense to use % error to compare models from different time periods on different subsets of countries.
It makes no sense to use % error to compare models from different time periods on different subsets of countries.
Last week, IHME added an additional 50+ countries to their projections, despite not able to beat the baseline for existing US states and countries.
During a time when the world looks to the US for leadership in the sciences, their approach is reckless and irresponsible.
During a time when the world looks to the US for leadership in the sciences, their approach is reckless and irresponsible.
Two examples of questionable current IHME forecasts:
Egypt: 80 -> 1,600 deaths/day by Sep
Turkey: 20 -> 700 deaths/day in Oct
It'd make more sense to make sure that the existing model can make reasonable forecasts before expanding to new countries.
Egypt: 80 -> 1,600 deaths/day by Sep
Turkey: 20 -> 700 deaths/day in Oct
It'd make more sense to make sure that the existing model can make reasonable forecasts before expanding to new countries.
Furthermore, new data is especially critical at each stage of the epidemic. Hence, we do daily updates to make sure new information is quickly captured.
IHME has more than enough resources to do more frequent updates, but it's clear they are focusing their efforts elsewhere.
IHME has more than enough resources to do more frequent updates, but it's clear they are focusing their efforts elsewhere.
I should also mention that the baseline I used is the simplest baseline possible.
In practice, if IHME were to use 12+ data sources / input variables, they should expect to perform better than a model that uses only a single data source.
In practice, if IHME were to use 12+ data sources / input variables, they should expect to perform better than a model that uses only a single data source.
Meanwhile, here are five models that we have identified to be able to consistently beat the baseline (for US forecasts):
Lastly, our evaluation is open source, so anyone can verify any of the data we presented as well as find other evaluation dates/periods. The pattern from above is fairly consistent:
github.com/youyanggu/covi…
github.com/youyanggu/covi…
