Thread by @ojwatson92

Our analysis on how official COVID-19 mortality has hindered our understanding of the COVID-19 pandemic globally, what novel approaches have been taken rapidly to fill the data gap and what long term solutions are required. A thread on the story behind this piece:

https://twitter.com/bmj_latest/status/1448589707600879621

Firstly, we wrote this article in October 2020 - 1 year ago almost to the day. No journals wanted it at the time - it never even went out to review. They weren’t interested in unreported COVID-19 deaths. They weren't interested in COVID-19 in lower income countries.

At the time, official COVID-19 deaths were being used for comparing the effectiveness of country responses to COVID-19 and everyone was stuck debating/arguing why some countries had “crushed” COVID-19 and why scientists were “baffled” by these differences.

These narratives were very unhelpful as they did not move forward our understanding of COVID-19. The political point scoring underpinning these hindered discussion about the reasons behind the discrepancies between early modelling results and observed COVID-19 deaths in LICs.

What modelling discrepancies? Here is one example below - these are from covidcompare.io and show how our model projections for India back in July 2020 were over predicting future epidemic trends in most lower income settings.

As the first wave was growing, and with mobility trends starting to increase, we projected that the epidemic would continue and more COVID-19 deaths would follow. This did not happen. In fact, mobility increased and reported COVID-19 deaths flattened before decreasing in India.

In response, we wanted to know what we had missed. A pattern was emerging - we were overpredicting transmission in lower income settings. We collected data to conduct ecological analyses to see if there was something that was driving these patterns - medrxiv.org/content/10.110….

This analysis tested hypotheses that had been circulating about the impact of other infectious diseases, other vaccinations, demographic effects, climate etc. Nothing significant came out. Largely because of data gaps. Under-reporting was still the most likely explanation.

So we needed a data set to understand under-reporting in lower income settings. But that was very hard - there were no estimates of excess mortality in lower income settings. Without this, it is hard to know a) whether deaths had been missed or b) if COVID-19 was less severe.

In Aug 2020, @ejbeals reached out with a report from Damascus. This was the first data I had seen that might help. Combined with an amazing citizen science initiative using obituary notes on Facebook we could show that 1% - 3% of deaths had been reported.

https://twitter.com/ojwatson92/status/1385679533844619265

After that study, I viewed official COVID-19 deaths very differently. This could explain why some epidemics didn’t take off in many parts of the world - they were just being missed. Estimates also from Aden and Khartoum showed large, unobserved epidemics earlier in the year.

These levels of under-reporting (<5% of deaths being reported) took me by surprise. However, they should not have - estimates of death reporting for other infectious diseases are similarly low for many settings.

These ideas, both on the parallels to other infectious diseases and the growing understanding of under-reporting, started to be discussed more widely. E.g. they were discussed in UK Government convened select committees during September 2020 - committees.parliament.uk//oralevidence/….

But why does this matter? Simply put, knowing the scale of under-reporting greatly changes our understanding of the pandemic’s dynamics.

For example, it will change how we quantify the effectiveness of previous interventions and will change where we understand the pandemic’s burden to have been greatest and where vaccines need to be prioritised.

Given these implications we wanted to get this simple message out there and we failed. Multiple journals were not interested - not even to send out to review. At this stage, I was deflated as I thought this was important to get out there and here’s why:

I previously thought data did not exist to understand under-reporting in LICs. It does - it just requires more work. The data is not as clean as the more readily available data from high income settings - settings that are viewed as more impactful in academic journals.

I had hoped that these observations would help spark other investigations into under-reporting and new methods to understand mortality and quantify how many COVID-19 deaths were being detected.

Unfortunately, that didn’t happen and by the end of 2020 I was mentally broken trying to get that message out. I had emailed more journals with pre-inquiries and had spoken to journalists but most people still wanted to focus on the belief that countries had been spared.

One example, this coverage of India COVID-19 being fundamentally different in Feb 2021 - I tried and failed to make my argument strongly enough for why I believed it was under-reporting - newyorker.com/magazine/2021/….

Then everything changed with India's 2nd wave. This was impossible to ignore. The coverage prompted tremendous investigative data journalism efforts (see @Rukmini and @muradbanaji) to find mortality data. These unequivocally showed massive under-reporting during the 2nd wave.

BUT they also showed large under-reporting on a similar scale proportionally during the first wave. However, because of the lower transmissibility, the flatter first wave resulted in excess mortality that was less striking and harder to detect against baseline mortality.

After this, excess mortality datasets just started appearing from journalists, citizen-led initiatives and hacker groups. They were also more frequently being used to evaluate the pandemic’s burden and prompted more reviews into unreported COVID-19 deaths.

For example, Peru conducted a review into undetected COVID-19 - they doubled their official death toll in June 2021 - aligning with excess mortality data from the country. bbc.co.uk/news/world-lat…

I honestly believe, that much of this is thanks to #WorldMortalityDataset. It is an invaluable resource and a testament to how important transparency and open access data is. Massive thank you to @ArielKarlinsky and @hippopedoid. Please read their paper elifesciences.org/articles/69336

With the collection of excess mortality datasets globally growing, further analysis could finally be done to start understanding the global patterns in COVID-19 dynamics.

For example, the Economist’s analysis is brilliant and uses this data to infer excess deaths worldwide, suggesting that currently over 16 million deaths have occurred worldwide. Over 3 times as many as reported. Thank you @Sondreus for all this work economist.com/graphic-detail…

Similarly, a recent meta-analysis of IFR in lower income settings was posted on medrxiv. This suggests the burden in developing countries is actually higher than in high income settings, with death registration obscuring IFR inference @GidMK. medrxiv.org/content/10.110…

The narrative was now changing and people were investigating under-reporting. So, I reached out to BMJ as they had published the first study looking at under-reporting in an African setting without excess mortality: bmj.com/content/372/bm….

And that is the story. Why spend 30 tweets detailing it? Because we will make the same mistakes next pandemic. Because, even if we are able to ensure complete death registration and certification, there will be political motives to ensure data is not made available.

And that is why we wrote this article:

1. To show the scale of under-reporting and why it changes our understanding of the pandemic and will continue to bias where we view the burden of the pandemic to have been greatest and thus most in need of interventions.

2. To encourage others to start looking for new datasets and alternative methods and engage with local researchers in these efforts rather than just dismissing data quality.

3. To emphasize the need for increased investment in civil and vital registration systems and call for supranational bodies to create the structures needed to depoliticise mortality and encourage data transparency. Without that, the next pandemic will play out the same.

Massive thank you to all the other authors - @arran_hamlet, @PatrickWalker8,@andradjaafara, @azraghani. Particular mention to @charliewhittak who wrote the most beautiful and compelling arguments in this piece. Truly grateful to all he has done.

Similarly thank you to all the other researchers and individuals looking into under-reporting and excess mortality worldwide. @Mahan_Ghafari @jburnmurdoch @helleringer143 @charliegiattino and many others.

And lastly, thank you to @SchmidtFellows for allowing me the freedom and flexibility to work on this area of research and have the opportunity to interact with as many wonderful and interdisciplinary researchers.