My Authors
Read all threads
How not to do a #COVID19 seroprevalence study
New paper out today from CDC that is so problematic I want to cry.
(great material for @callin_bull)
Quick thread.…
We've known since Feb(!) that only fraction of SARS-CoV-2 infections cause illness & less cause severe illness. This led to huge uncertainty & confusion about fatality of #COVID19:

& contributes to Qs about herd immunity:
Best way to measure all infections is seroprevalence study which estimates fraction of people w/ antibodies (Ab) from past infection. Method isn't perfect; e.g. a small % won't mount Ab & titres will wane but most (>90%) will be detectable still ().
The challenge is doing study properly. If you want to estimate the fraction of a population (& subgroups w/in it) that have been exposed you need a few key things. Most important by far is random sample of people. If you don't get this part of study right, data are poor.
(Note: I've been whining about this for months:
Note: Bias can go in either direction
You can get serop too high if you invite people to join study (e.g. w/ a facebook ad) & those w/ symptoms are more likely to join; also if you sample people who have high risk jobs.
Bias can be huge: 3.5x+
Compare, e.g. seroprev of hispanic & white people in NYC. If you didn't know you were sampling different pops you could get very diff answers.…
You can get seroprev too low if you sample people that have been avoiding high risk activities: people who can work from home; who can pay for grocery delivery; or who can choose not to leave house.
Unfortunately, MOST seroprevalence studies I've read have substantial bias & in most cases we can't assess amount of bias so we can't adjust study to get accurate picture. Examples: blood donors; hospital patients admitted for non-covid reasons; HCWs, grocery store customers, etc
So what about today's paper by CDC in JAMA (respected public health org, top medical journal)?
Amazingly bad.
Note: CDC previously wrote a book on how to collect random sample from communities (…) so we know they know how to do it & how important it is.
But did they implement this procedure? No. Instead they looked in the freezer of clinical labs to see what was left over & decided the dregs were good enough to inform the public during a pandemic.

No effort at all was made to collect a randomized sample.
Really CDC?
Did they make a herculean effort to address potential biases by pulling dozens of variables on samples that might make up for sampling biased pile of tubes left in freezer? Nope. None at all. Only adjustments were for test characteristics, age, sex.
How much does this matter? A ton, it appears. A quick peruse of the results from the paper makes it clear that the data are essentially uninterpretable.
How about some data on a topic no one cares about right now: safety of school re-openings. (clarity note: sarcasm)
Study shows age breakdown for 10 locations. Some might initially get excited & see data to supports their arguments about children being less susceptible, but...
Look carefully! There's patterns here for everyone!
IN LA, NYC, PA serop in 19-49yr are 3x kids (0-18). 6x in CT!
But in MN, kids 0-18 have HIGHEST serop by a ton! Also FL 0-18 2.5x 19-49. SF 1.6x higher.
Am I cherry picking? You bet!
Aren't they all good data b/c CDC, JAMA?
Nope! Besides obvious & apparent issues (small N for many age groups/sites so huge CIs), there are unmeasurable biases.
I know of no reasonable biological reason that serop would be much higher in kids in 3 places but lower in 4. But I can imagine many BS reasons!
There's only one way to safely use these data: toss them in the circular filing bin. (for clarity: trash)

If the patterns fit your pre-conceived ideas, they are reasonably robust.
If they contradict it, they are problematic.
Why collect bad data in the 1st place?
Answer: Because it is easy.
Instead of designing proper study w/ randomized sample (see e.g. ), they literally just took blood samples they already had lying around.
Also, one bonus gift for you...
Winner for most misleading graph:
Did you know that over time fewer people have antibodies? Waning Ab! (joke)
That's what this graph seems to show. @callin_bull this one is for you!
There are so many ways this could have been presented in informative way. This isn't one of them.
Are the authors aware of the limitations of this study? You bet they are! They lay it out pretty clear in the Discussion. But if they have to write this in the Discussion doesn't it make you wonder why they did the study in the 1st place?
The only thing worse than this type of study is doing it again, and guess what? That's what they're doing. It's a recurring study! 2nd round has already been done & is ready for mis-interpreting (have at it!):…
Dear CDC,
Please read the book you wrote on randomized sampling & conduct a study that would actually be useful in helping us understand this pandemic.
Missing some Tweet in this thread? You can try to force a refresh.

Keep Current with A Marm Kilpatrick

Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

Twitter may remove this content at anytime, convert it as a PDF, save and print for later use!

Try unrolling a thread yourself!

how to unroll video

1) Follow Thread Reader App on Twitter so you can easily mention us!

2) Go to a Twitter thread (series of Tweets by the same owner) and mention us with a keyword "unroll" @threadreaderapp unroll

You can practice here first or read more on our help page!

Follow Us on Twitter!

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3.00/month or $30.00/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal Become our Patreon

Thank you for your support!