Thread by @ewanbirney

Gather around again for a tweet-thread about "models", their types, what they are (and aren't) and how to use and comment and/or critique them.

First off, models are always ways of understanding the real world. The well worn adage, "all models are wrong, some models are useful" serves you well, and focuses on *utility* of models rather than correctness.

Already you can see you might change what is useful in different settings - indeed, there are some very practical constraints about what things one can or cannot do because we have both time and decisions - this is a system with *lots* of endogenity as the economists would say.

In COVID (and many other things) one can distinguish 3 types of models. One is "historical" or "explanatory" models - writing down how you think the world works, and using only retrospective data showing how it fits or doesn't. Often you might have more than one type of model.

(one doesn't need a time dimension for this, hence explanatory, eg, a model for how DNA, which is sequenced with errors, makes RNA which splices and is translated into protein is a model I wrote down in my PhD. No time in this case; different models worked better or worse)

For explanatory models one can, for example, use death data to model earlier infection data, ie, dieing of COVID requires one to be infected by SARS-CoV-2. So one can bring different data streams together.

Use these models to understand properties of the system; eg, vaccine effectiveness or transmission properties in different levels of lock down stringency or the impact of human genetics on progressing for severe COVID.

The second is "forecast" or more relevantly for COVID "nowcast" - nowcast being "what is the level of infection right now". These models are present - they are very accurate for 3 or 4 days with steady data (sadly not holidays) + one can push them with "momemtum" assumptions out.

The nowcasting models are not widely distributed in my experience (I do like the twitter based people who do this - eg, @PaulMainwood), and as it happens for steady data in the settings I know best (UK, France...) the data itself is usually better than nowcasting

This is particularly true of the UK's ONS survey which lags a bit behind test data but does not suffer from endless arguments about testing behaviour change or other things. From this perspective, case data is a now(ish) cast of ONS data.

The final set of models in COVID are scenario models - these are "What If" models - "what if we didn't change anything?", "what if Omicron was as severe as Delta, or half as severe, or 1/10th?".

These are the models that generate the most angst and debate precisely because they go into an important set of decisions to make.

Scenario models are not trying to forecast; they are trying to model the impact of decisions - they need good data (tick in many countries) and "good enough" models, validated by explanatory modelling.

A good example here is how one handles seasonality - do you put it in as a term, or leave it out but have a behavioural term which picks up the "being more inside" - here one can test different explanatory models and decide the best framework for a scenario model.

Frustratingly the scenario models have been narrowly focused on infection and hospitalisation - in some fantasy world they would be one component to scenario models of NHS operations, which has to plan ahead as well, *and* some whole economic model as well, ideally CBA like.

This is "fantasy uber modelling", and probably not practical, but just as scenario modelling helps inform decision making, just trying to bridge these worlds would (in my view) help understand the landscape better. One for... future pandemic prepardness. Not for now.

Now - how to critique these models? Explanatory models are about how useful they are for explaining the world. A key question for these models is how much support can you give to this causal structure of the world, ie, does doing X really shift the outcome probabilities of Y.

When this is true, people can leverage this in decisions - a great example here are "do masks work" which one can try to do via explanatory models or, when one can set things up, by RCTs (often requires explanatory modelling)

Forecast and nowcast models are in someways the easiest to critique - did the forecast fit the real outturn? Simples.

Scenario modelling critique is really "did it help decision making". Which is ... very much in the eye of the decision maker. For COVID scenarios I personally don't worry at all about the timing of the outturn, but the shape and amplitude.

Amplitude for most of these models are best in log() space (as is, interestingly, a fair bit economics/markets as well). Frustratingly of course healthcare and impact is in the linear space - small shifts in the log space can be awful in linear space.

The shape of course is the easiest to see. For example, over the summer/autumn in the UK of 2021 the average of the scenario models was well in line with outturn (in log space showed this best) ...but... all scenario models had a peak sometime, which didn't occur.

(personally I think this was because we started have broader access to testing, and when *local* infections picked up, people tested more, meaning there was a local feedback loop. This is though an explanatory explanation that should be tested. As stated, it is just a narrative!)

As well as the models themselves, the people who design, critique and roll out the models themselves often know their weak spots and their expert judgement is one other aspect to feed in. Other people (some on this site) critique them well, hence transparency is key.

So - forecast models are the easiest to understand and critique, but just one type (and not widely used in COVID). Explanatory models most people have as "doing science". Scenario models are to help decision makers.