Thread by @Loretta_Torrago

While @ProfEmilyOster denies consulting with @GovRonDesantis, she hasn’t yet repudiated her study. She should. The study claims to show mitigations don't work but has so many design flaws, even effective measures would fail her trials. Very long🧵.
nytimes.com/2021/06/22/us/…

Probably because they’re assumed to be the “gold standard”, Oster’s studies employ the all the signs of being Randomized Control Trials (RCTs) in which mitigations: mask mandates, in-person density and ventilation are compared with schools absent those same mitigations.

Oster’s conclusion is, more or less, that mitigations make no difference to Covid case counts in students, staff and teachers associated with schools.

An RCT is an experiment in which something, referred to as an intervention, is tested to see if it works or not.
(R) Randomization is important to a study because, without it, the success of an intervention might be due to confounding factors and not due to the intervention.

(C) Control groups are also important because they provide the contrast between those who had the intervention and those who did not. In a drug trial, for example, the drug is the intervention.

The control group is the group that does not get the drug being tested. The drug’s effectiveness is derived from a comparison of how well the intervention group fared in comparison to the control group.

If the drug is effective, say at lowering blood pressure, my intervention group’s blood pressure will be lower than the control groups.

But an effective intervention can look ineffective if the trial isn’t done correctly: if the control group gets mixed up with the intervention group, if outside factors impact the result, or if the groups aren’t randomized. All of these things, and more happen, in Oster’s study.

1. Are the Schools Random Samples?
Covid case counts and mitigation practices were gotten from publicly available resources or, in the case of mitigation, occasionally from the school itself.

Nonetheless, the schools Oster includes in her studies are schools that opted to participate in her data gathering dashboard. This has consequences for randomization. vox.com/the-highlight/…

While Oster’s dashboard may have grown, the fundamental fact remains that schools with the money to implement mitigations are more likely to opt in than schools without resources to mitigate.

That’s a problem, not merely because randomization per se is important, but because we already know that Covid has less of an impact on the well-off. (info.primarycare.hms.harvard.edu/blog/social-co…).

This matters especially because Oster’s study isn’t a study of in-school transmission but of Covid rates amongst students, faculty and staff whether or not they got Covid in schools.

As such, Covid outcomes may be more a reflection of the community’s ability to be protected from Covid, which of course includes the school, but is certainly not limited to it. This point is conceded.

What this means though is: we are looking for the differences mitigation practices make to Covid outcomes amongst a group that Covid has made the least difference to. In that case, differences are going to be harder to find.

A. Mask Mandates
According to Oster, mask mandates make no difference to outcomes when compared to schools without mandates.

Here the intervention is the mandate and the control group, absent the mandate, is clearly not using the intervention. Nonetheless, it would be foolish to assume that merely because there is no mandate, there is no mask wearing.

Likewise, it would be foolish to assume that a mere mandate guarantees mask wearing compliance. This point is conceded.

More importantly though, many schools where masks are mandated provide mask breaks and, naturally, no kid wears a mask at lunch which makes the mandate school, at least temporarily, mandate-less or, in other words, lacking the intervention.

This matters especially because indoor dining has been implicated in a number of outbreaks (cdc.gov/mmwr/volumes/7…)
as has loud talking (pnas.org/content/117/22…)
both of which happen in school cafeterias.

So it’s not just that the intervention, the mandate, is sometimes lacking in the intervention group, it’s that it’s lacking precisely under the circumstances where Covid transmission is most likely to occur.

Of course, the mandate-less kids eat lunch too but, given Covid’s erratic transmission where over-dispersion dominates, (theatlantic.com/health/archive…) it may matter a great deal to outcomes that interventions are lifted at all within the alleged intervention group.

But we might look at whether mandates work in a different way; by looking at what happens when they are lifted. There we can see the impact, in terms of in rising cases.

B. Ventilation
Oster's study also considers whether schools made improvements to ventilation and looks for a difference in Covid cases between schools that made improvements and schools that didn't.

The only problem is: we don't know what the schools are improving from or to & what the schools that didn't improve started with.

In that case, we might be comparing a school that didn't improve it's ventilation because it had an excellent HVAC system, to one that did improve ventilation but went from one very bad system to one only slightly less bad by say, cracking a window.

Yet the open window school would be in the intervention group while the excellent HVAC school would be in the control, despite the fact that, the control group actually has better ventilation than the school in the intervention group.

What’s worse, we might have, in the control group and the intervention group, the very same ventilation system if, for the intervention group, this represents an upgrade whereas for the control group it is a continuation of the system they had previously.

C. Density
Oster’s study claims higher case rates are associated with lower classroom density.

The explanation Oster offers is that, with schools closed, students are likely engaging with other groups and in extracurricular activities.

However, Oster gives no evidence these outside of school activities are less safe than school. In fact, a far more plausible alternative is that when case rates are high, kids are home because concerned parents are keeping their kids home rather than sending them to school.

That, at least, is consistent with studies that show crowding and lack of distancing contributes to transmission. jamanetwork.com/journals/jama/…

What’s more, while the study takes into account enrollment, it does not take into account actual attendance which is important because sparse attendance can serve as proxy for distance and hence density.
academic.oup.com/cid/advance-ar…

Further, density, as judged by enrollment, fails to account for staggered schedules, with a subsequent overestimation of actual classroom occupancy with the same result: fewer students at a time creates a less dense classroom; a fact hidden by high enrollment numbers.

A further complication arises in the assessment of density and Covid outcomes when Oster ignores the interplay of ventilation and density and that a properly calibrated ventilation system can compensate for increased density by increasing air exchange rates.

In which case, appropriate ventilation that compensates for density approximates the work of decreasing density. IOW, a factor (ventilation), aside from density, is impacting the result making it look like density doesn’t matter (but because ventilation is doing the work).

D. Outside Influences
Failure to have well-controlled intervention and control groups is problematic enough but perhaps the most important flaw in the study is the one that impacts the study the most: it’s not a study of in-school transmission.

Consider, for example, how that plays out in the case of mask mandates. Suppose a kid from a masks-are-required school wears a mask in school. If that kid, outside of school is maskless & contracts Covid, it’s hard to make the case that this cuts against the mandate.

After all, the function of a school mandate, like any school rule, is to control what happens in-school. Likewise, for ventilation and density.

Now one might think this doesn’t matter if what is being tested is the impact of in-school interventions on the community. But, importantly, that is not what is being studied. What is being studied isn’t the community, but the barest subset of it: students, teachers and staff.

And we know, from many studies, that schools impact the community transmission rates in significant ways as study after study has shown that when schools close, cases go down and when they open, cases go up.

https://twitter.com/Loretta_Torrago/status/1381264249142534146

Emily Oster’s study looks for all the world like an RCT. But if it is, it’s a deeply flawed one and the flaws are exactly why she gets the result that mitigations make no difference. The problem is the design; not the mitigations. medrxiv.org/content/10.110…