Thread by @youyanggu

The COVIDhub Ensemble model that combines all the models did not perform well over the past 2 months.

This is due to the fact that the majority of model submissions did not properly forecast this current wave.

Roughly half of all models failed to beat the baseline.

This is a known issue with pandemic modeling. For most scenarios, it's beneficial for models to make forecasts close to the status quo (since that's usually true).

This means the they're accurate a majority of the time, but they will miss large spikes such as this current wave.

On the flip side, if a model predicts a large spike and is wrong, it will be heavily penalized by most evaluation metrics. This can happen even if the spike does happen but is a few weeks early/late.

That's the dilemma a lot of modelers face, including myself earlier this year.

Here are the most accurate models in forecasting deaths and cases from October-December:

Top models for deaths: UMass Amherst (@reichlab), Dean Karlen, USC (@ajitesh47 et al)

Top models for cases: LNQ (Russ Wolfinger et al), Dean Karlen, DDS UT-Austin (Mingyuan Zhou et al)

These are based on my own evaluations for 4 weeks out. You can find the full evaluations here: github.com/youyanggu/covi…

You can find more information about each team here: github.com/reichlab/covid…

So what do these top performing models tell us about deaths and cases over the next 4 weeks?

Deaths will continue to rise to ~3,000 per day.

Cases may flatten at 200,000 per day, or continue to increase past 300,000 per day.

The takeaway? Most models are not useful. Even the most useful ones are not always accurate.

Models are not meant to be crystal balls. It's important that we understand the limitations of what models can & cannot do.

When used correctly, they can be powerful tools.