@trvrb I should take 12 hours away from work more often. I had thought about reading this study carefully last night and commenting, but slept instead. When I woke up I found @nataliexdean @trvrb and @jjcherian did most of the work. I agree with all the statistical points of these 3.
@trvrb @nataliexdean @jjcherian I'd add that some of the data gathered but not in this initial report are important for their own sake and could help address some of the concerns.
@trvrb @nataliexdean @jjcherian Two key data elements not reported here are 1) the symptom reports from those testing + and - and 2) the crosstab of IgG+/IgM+. Test positive seems to be defined as positive on either, but relevance of the company's and Stanford's validation expts would be clearer if broken down
@trvrb @nataliexdean @jjcherian For example if the mix of ++,+-, and-+ among the "test positives" is similar to that in the validation studies, then they are probably more interpretable than if not.
@trvrb @nataliexdean @jjcherian More importantly, the concern raised by @nataliexdean that sick people are more likely to come in to get tested is not only major but also perhaps addressable with these other data elements.
@trvrb @nataliexdean @jjcherian If a large proportion of the positives are symptomatic at time of test and/or are IgM+/IgG- then this concern would be very much warranted. If the opposite (asymptomatic and/or IgG+/IgM-) then the concern would be less warranted. Strange that these are not reported.
@trvrb @nataliexdean @jjcherian Hoping that peer reviewers will require these data to enhance interpretability of the study.
@trvrb @nataliexdean @jjcherian A word on the sociology of this. Some commentators have remarked on scientific polarization. In a sense this is true. For example I have disagreed publicly with Ioannidis and would have also with Bendavid's WSJ writing if I'd had time.
@trvrb @nataliexdean @jjcherian In policy disagreements it's key to bring evidence to support one's views, and if it's necessary, generate evidence on areas of controversy. Kudos to them for generating such evidence and providing one interpretation of it (which supports their "it's overblown" view)
@trvrb @nataliexdean @jjcherian The great thing about science is that we are trained (think student journal clubs) to find every flaw we can in others' data and interpretation (ideally that training is designed to help us find and fix flaws in our own work before publication).
@trvrb @nataliexdean @jjcherian It is a mark of respect for someone's scientific work to take it seriously enough to look hard for flaws. Of course this can be done politely and in the case of trainees supportively, etc. It's not a fight, it's a process of strengthening inference.
@trvrb @nataliexdean @jjcherian Anyone who doesn't want to be told from time to time every possible reason why they might be wrong about something is going to be unhappy in science (unless they become the decider on large amounts of grant funding, at which point most people stop disagreeing with them).
@trvrb @nataliexdean @jjcherian So what all of us are doing in critiquing this paper is a mark of respect. The (mild) polarization at this stage may in fact be constructive, because it is often easier to see flaws in an argument when you disagree with the conclusion.
@trvrb @nataliexdean @jjcherian In the end, no one study is going to answer the question of how badly we are underascertaining cases/overestimating IFR. A number of studies with different technologies, analytic approaches, etc., will be needed. The facts will out.
@trvrb @nataliexdean @jjcherian Finally, a note on the comparison to seasonal flu. This is not apples to apples. If we counted deaths from seasonal flu the way we count deaths from COVID (requiring a positive PCR) we would get much less than the 20-50K or so per year that the US typically has.
@trvrb @nataliexdean @jjcherian To walk through the CFR calculation for flu (in orders of magnitude): US population ~3x10^8. Annual flu infections ~10%=3x10^7. Annual flu-attributed deaths ~3x10^4=0.1%.
@trvrb @nataliexdean @jjcherian Of those 30K deaths (again in orders of magnitude) we estimate that 1/7 have "pneumonia or influenza" on death certificate; main other causes are non-P&I respiratory, and circulatory (MI, stroke). ncbi.nlm.nih.gov/pmc/articles/P….
@trvrb @nataliexdean @jjcherian ncbi.nlm.nih.gov/pubmed/12517228 gives a similar picture in an earlier analysis from CDC (some statistical problems with the regression there, to be clear, but same qualitative finding).
@trvrb @nataliexdean @jjcherian Because flu testing and death certificates are not linked, we don't know if any of those MI stroke chronic lower respiratory etc. deaths attributed to flu statistically tested positive. Probably very few.
@trvrb @nataliexdean @jjcherian So these 30K attributed deaths from flu each year, if we ascertained them like we're ascertaining COVID deaths, would be more like 4-5K per year (the P&I fraction). Bottom line a more apples-to-apples comparison of confirmed COVID CFR to seasonal flu would be compare to <0.02%
@trvrb @nataliexdean @jjcherian Finally, as others have pointed out on twitter, CFR is only part of the story: if ~10% get infected with flu per year, and everyone is susceptible to COVID and we get close to herd immunity threshold over some period, that is at least 5x more infections than a year of flu.
@trvrb @nataliexdean @jjcherian One more thought on the Santa Clara paper: Peer reviewers should ask for the recruitment ads. If this tweet accurately describes them this is a real problem (which might not be spotted by my suggestion to look for current cases by overrep of IgM)
