Thread by @drStuartGilmour

The #economist is publishing a regularly-updated model of excess deaths due to COVID, which is being used to tell just-so stories about countries the Economist doesn’t like. Let’s talk about the dangers of using machine learning for analysis when you don’t have data.

Before we do, let’s just take a moment to appreciate the deep racism of this picture from a report using the Economist's numbers by the Centre for Strategic and International Studies. When you see a cover pic like this, you can guess what the contents are gonna be.

Having seen who is using this data and hazarded a wild guess at why, let’s look at some results. First, let’s compare the UK and Vietnam. According to the Economist, Vietnam had nearly as many excess deaths due to COVID-19 as the UK, despite containing the virus for a year.

Since the 1st August 2021 when Vietnam’s outbreak began, it has never once had more daily cases than the UK. Yet this country with this epidemic experience is supposed to have had as many deaths as the UK.

The economist believes China has had as many excess deaths as the USA. This is simply not credible. It’s not possible that China would have an epidemic on that scale and nobody would know.

This data is simply wrong. So how is it possible the economist got it so wrong? And why do they publish anyway? The answer to this question is also relevant to other projects that aim to estimate health data where that data is missing, like the #globalburdendisease studies.

The economist have limited data sources – they use complete mortality data from the human mortality database and the world mortality database, but these only cover about 110 countries. They don’t have anything for China or Vietnam, for example.

They then build a massive model based on this data to estimate what the true excess mortality might be using a machine learning method called gradient boosting. They predict deaths in other countries based on the countries with data.

For example, using data on covid tests per person, positivity rates, human development indices etc they can guess what deaths might be in countries without detailed death data. Gradient boosting selects the “best” variables and their effectiveness.

This has a huge and simple problem: you can’t apply relations that exist in countries with full pandemics to countries where COVID is contained. A model can’t tell that the low testing rate in China is because of no cases – it will assume under-testing.

They seem to have included some index of political freedom from “Freedom House” but failed to include any variable indicating whether the country *actually has a COVID epidemic*. Hence, the model assumes China and Vietnam are under-diagnosing. This is a bad and simple error.

China does not have half a million excess COVID deaths, Vietnam doesn’t have 100k, lots of Asian countries are doing better than the USA, and fancy models that serve to reinforce western wishful thinking don’t help. See also the IHME’s estimates of COVID deaths in Japan.

There are many other problems with the Economist’s work. First, they seem to apply a linear regression model to deaths, rather than over-dispersed Poisson regression. This allows the prediction of negative deaths in a country!

Next, they treat weeks and months as “fixed effects”, a weird obsession within economics, rather than fitting a spline or some other model that is less arbitrary. This is not how we do this in epidemiology.

They calculate excess deaths as a difference from the predicted deaths, even though deaths naturally change every year. They should generate uncertainty ranges around the predicted deaths for 2021, and then report excess/exiguous deaths *over or under those limits*.

This is ironic because they reference a BMJ paper of excess deaths in Hubei which tells them to do all of these things (except the prediction limits). There is a method for excess deaths, called the Farrington method, used by the CDC. But they ignored it!

If you want to see how excess deaths are actually calculated (when data is available), check out papers by some of my colleagues and I that find limited changes in Japan. doi.org/10.1016/j.psyc…

A general problem with these big multi-country studies that try to fill in data for countries they can’t acces is that they use data-driven methods to try and infer missing information. It’s at the core of the global burden of disease (GBD) study and it’s dangerous.

GBD estimates of depression prevalence in Africa, for example, can be based on a small number of studies in a handful of countries to generate evidence on the whole continent. We should be very careful about this process.

Also, wherever possible these multi-country studies should incorporate data and collaborators from every country they include. The GBD studies do this; the Economist clearly have not. Their conclusions are not reliable as a result.

To summarize: The Economist is doing a shoddy version of work that has already been done by experts in epidemiology, and conveniently confirming the suspicions of western ideologues that those shifty Asians are hiding the truth, and COVID-19 is really bad over here.

It’s not as bad over here. Western governments failed on COVID-19, and no amount of dodgy modeling and just-so stories by their mates in the dismal “science” will let them off the hook for their criminal response to this challenge.