Health Nerd Profile picture
14 Oct, 39 tweets, 14 min read
John Ioannidis, of "Most Published Research Findings Are False" fame, has now had his paper on IFR published

Let's do one, final, twitter peer-review on the study 1/n
3/n I should say at the outset here - the only personal comment I would like to make about Professor Ioannidis is that he is a very smart man who I respect tremendously

I will, however, examine the paper, because I think that is what science is all about
4/n At first glance, and indeed on deeper reading, it is clear that very little has changed from my previous looks into the paper
5/n The methodology is still the same, and the eventual conclusion remains that the median IFR of COVID-19 is 0.27% (originally he estimated 0.26%)
6/n The author then concludes that the IFR "tended to be much lower than estimates made earlier in the pandemic", which is odd because his own estimates made earlier in the pandemic (in May) were...lower
7/n Indeed, as we can easily see, the resulting low IFR is simply a consequence of the low quality of the review itself and has very little to do with when the estimates were made
8/n For example, the review does not adhere to PRISMA guidelines (the most basic recommendations for reviews of this kind) which is very strange given that Prof Ioannidis himself is a co-author on the original PRISMA statement
9/n This has lead to a problematic situation, where there is no rating for study quality, publication bias, and indeed little consideration in the manuscript for how the quality of the published evidence might impact the review
10/n As we pointed out in our systematic review and meta-analysis of COVID-19 IFR, this is an issue because higher-quality studies tend to show a lower seroprevalence and thus a higher IFR
sciencedirect.com/science/articl…
11/n (Interestingly, Ioannidis cites our study but gets the numbers wrong, in what is distressingly something of a trend in the paper generally - we actually estimated 0.68% in the published paper which came out recently)
12/n We can see the issue with non-adherence to PRISMA in the methods section. These are clearly not the search terms used, as entering them into these databases results in 100,000s of results
13/n There are also still clear numeric errors remaining from previous versions of the study. For example, this number from a paper looking at people going to hospital in New York should read 44%, and not 47%
14/n And there are new errors as well. In this study of blood donors in Rhode Island, the authors estimate a seropositivity of 0.6%, while the review paper has 3.9% instead
15/n But by far and away, the biggest error in the text is simply to do with using clearly inappropriate samples to estimate population prevalence

This is a fundamental flaw in the paper, and really something of a basic epidemiological mistake
16/n Some of these studies are just so clearly inappropriate to infer a population estimate that it doesn't really require explaining. Samples of a single business in a city, or inpatient dialysis units
18/n Then we have blood donors, who again may give an erroneous result. These are people who, DURING A PANDEMIC are happy to go out and about and give blood. It is quite possible that they are MORE likely to have been infected than the general population!
18.5/n There are also a lot of included studies from places in which there is almost certainly an enormous undercount of deaths

For example, India, where the official death counts may represent a substantial underestimate bmj.com/content/370/bm…
19/n A very basic, reasonable thing to do would be to conduct a sensitivity analysis excluding these biased estimates, to see what happens when you only use representative population estimates

Which we can do
20/n If we take the median of only these somewhat good-quality studies (some of them still aren't great, but at least they're not clearly inappropriate), we get a value of 0.5%

Double the estimate of 0.27%
21/n I thought at this point I'd briefly look at blood donor studies, because they are an interesting case study

The author argues that these should be included because, due to "healthy volunteer bias", at worse any estimate should bias the IFR results upwards
22/n Well, we can now actually test this theory and see if it is true. Enough studies have been done that we have COVID-19 seroprevalence estimates from BOTH blood donor studies AND representative samples and compare them
23/n For example, in England an ongoing study on blood donors by PHE estimates that 8.5% of the population has developed antibodies to COVID-19

However, the ONS with their massive randomized study puts the figure at 6% instead
24/n In Denmark, a robust population estimate put the figure at 1.1%, while their blood donor study estimates 1.9% have been infected previously
25/n Indeed, in every location where both a non-probabilistic, convenience sample has been taken (not just blood donors) AS WELL AS a well-done population estimate, the convenience sample overestimates the seroprevalence
26/n We have a new paper that we're working on that suggests that using such estimates will usually overstate the true seroprevalence by a factor of about 2x

Which means the true IFR would be double the number computed from such studies
27/n There are also still, after many revisions, studies that have been excluded inappropriately from the estimates

This study from Italy, for example, which produces an estimate of 7% (!) for IFR in the region
29/n Similarly, there are numerous country-wide efforts not looked at in any way, such as the large population studies conducted in Italy (150,000 participants) and Portugal (2,300 participants)
30/n And while there is a very brief discussion of the variation in IFR by region, the main component (age) - as we have demonstrated - was barely addressed, with the author instead focusing on vague speculation about healthcare systems medrxiv.org/content/10.110…
31/n We can actually see how age of those infected impacts IFR quite neatly from some of the studies in this review

Qatar (0.01%) and Spain (1.15%) look very different, right?
32/n Wrong! In fact, the difference here is entirely explained by age!

In Qatar, infections have mostly been limited to the immigrant worker population (<40 years), with this group representing more than 50% of infections
33/n Since this group is at a very low risk of death from COVID-19, the population IFR is MUCH lower than in Spain, where infections among the elderly have been much more common
34/n All of these errors are a shame, because to a certain extent I agree with the author

IFR is NOT a fixed category. In the metaregression linked above in the thread, we demonstrated that ~90% of variation in IFR between regions was probably due to the age of those infected!
35/n Unfortunately, Prof Ioannidis appears not to have read this study, but if you are interested here is the preprint version to peruse medrxiv.org/content/10.110…
36/n Anyway, there are numerous errors remaining in the text that I haven't pointed out, but if you've reached this far in the thread I'm sure you're tired of me telling them to you straight up. Have a really careful look and see if you can find them!
37/n (As a start, there is now a representative population estimate from Wuhan out that implies an IFR SUBSTANTIALLY lower than the ones inferred in this paper from samples including hospitalized patients)
38/n Regardless, the main take-home remains, unfortunately, that this paper is overtly wrong in a number of ways, it does not adhere to even the most basic guidelines for this type of research, and thus the point estimate is probably wrong
39/n Sorry, typo in tweet 37 - should read an IFR SUBSTANTIALLY *higher*, not lower. The SEROPREVALENCE is lower (at ~2%) which implies an IFR of ~1.2%
40/n Oh, on an unrelated sidenote, it's quite funny that the author spends some time arguing that using a median is more appropriate than doing a R-E meta-analysis (as @LeaMerone and I did), so I quickly calculated the median for our study and it is higher at 0.79% for IFR 😂

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with Health Nerd

Health Nerd Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @GidMK

15 Oct
Something that I think is worth explaining:

The term "herd immunity" has a few different meanings, all of which are quite precise. This is an important point Image
In a very strict mathematical sense, herd immunity is the slowing of disease spread due to sufficient infected people in a population (such that Rt<1)
This leads to the second definition, which is the ability to calculate immunity within a population based on dynamics of disease spread
Read 9 tweets
13 Oct
Imagine if someone was to say "COVID-19 hasn't caused most businesses to close, they were all doing badly anyway"
"Lots of businesses close every year, how do we know that they didn't just close WITH COVID-19 rather than FROM it?"
"All this economic impact is overblown. My friend's cousin's third ex-wife lost her business, but it was in the red anyway"
Read 6 tweets
12 Oct
Have lockdowns been bad for the economy?

A new report from the International Monetary Fund has a very interesting answer to the question 1/n
2/n The report is here, and well worth reading. At the outset I should say that ALL OF THIS EVIDENCE IS UNCERTAIN

The IMF is great, and has tried very hard to be accurate, but it is important to take care when reading these findings imf.org/en/Publication…
3/n The basic summation is simple:

- lockdowns probably cause economic harm
- large outbreaks of COVID-19 also cause harm
- it is hard to disentangle this complexity
- there are almost certainly situations in which lockdowns are beneficial to the economy
Read 18 tweets
12 Oct
Recently, John Ioannidis, of "Most Published Research Findings Are False" fame, published a commentary piece on COVID-19 and global action

I thought it would be good to do a bit of peer-review on twitter 1/n
2/n You can find the paper here - it is a classic commentary piece, which means it is mostly the perspective of the author: onlinelibrary.wiley.com/doi/10.1111/ec…
3/n Given the author's very strong public stance since early March, it is perhaps unsurprising that the main message of the piece appears to be that COVID-19 is not very bad but government actions are
Read 39 tweets
10 Oct
It's fascinating what turns out to work in terms of helping people if you ignore empty moralizations about what we should and shouldn't do

This is but one example
For example, there's good evidence that an effective treatment for heroin abuse disorder is prescription (safe) heroin

bmj.com/content/327/74…
Safe injecting spaces - while counter-intuitive because you're assisting people in taking drugs - are arguably the most effective way to reduce the harms of some illegal drugs

ncbi.nlm.nih.gov/pmc/articles/P…
Read 6 tweets
8 Oct
Apparently, there is a fly on Pence's head

I am not watching the debate
Every person in the U.S. simultaneously googles "Is fly on head a sign of COVID-19?"
Oh hey look
Read 4 tweets

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!

Follow Us on Twitter!