Science writers: can you please take pause in getting story out to make sure that it won't be an embarrassment and damaging 1hr later? Also, please don't report on studies w/ no preprint. Those aren't studies. They are uncurated doodles. Wait for at least a 1st draft (preprint).
e.g. Serological studies are extremely useful when done well & extremely misleading when done poorly. #1 issue: recruiting bias. If biased for factor you can't adjust for after by comparing to target population (e.g. prev. covid symptoms), data may be uninterpretable and useless
All US serosurveys I've seen so far have tiny # of seropositive individuals (e.g. Stanford: 50; LA: ?? 863*4.1%=35 but is 4.1% raw or adj? Boston ~60); a small bias in recruitment could alter results 10x; none of these studies have been done w/ random sampling.
Obtaining random sample can be challenging. e.g. 1 Option: Contact random subset of ALL addresses (requires data), randomly select w/in household and re-weight w/ measured household size. Need refusals to be low % and unbiased in key metrics (may be hard to demonstrate).
Checking afterward that sample matches target population in key traits and limiting generalization to that target population is key (e.g. Option above excludes homeless, tourists/visitors/individuals who don't have addresses/receive mail).
For covid serosurveys exposure likely varies w/ profession (remote vs essential workers; high contact/risk jobs like Drs, nurses, pharmacists, cashiers), age, socio-economic status. Desire to participate varies w/ recent symptoms. Illness differs w/ age, pre-existing conditions.
Any study that does not successfully randomize (and ideally measure to check vs target pop!) these variables in sample may be biased; bias for some factors can be large and qualitatively change conclusions.
A greatly underappreciated difficulty is estimating prevalence when prev is low. Abs error in % is small but relative error can be enormous. Simple example: 1/1000 vs 2/1000; abs diff is 0.1%; relative diff is 2x. When prev is being multiplied by large pop rel diff matters!
e.g. in Stanford study raw seroprev 1.5% (50/3330). If bias in recruitment of people (e.g. those w/ previous covid symptoms) increased real # w/ antibodies from 10 to 50, abs diff is small 1.2% (1.5% vs 0.3%) but implications huge. 50-85 multiplier of cases becomes 10-17.
Given the need for solid methods and clear evidence of poor approaches being used, perhaps people who do this frequently could provide methods for (quickly?) obtaining random sample when, e.g. complete address lists aren't available? @EmilyGurley3 @michaelzlevy @Justin_Remais
