Thread by @Mark_Ungrin

Happy to announce that our long-awaited preprint is up!

We assessed shortcomings in the Loeb (2022) clinical trial paper that reported medical masks to be noninferior to N95s for COVID prevention.

It's a long read, but we hope you'll find it educational.

As usual for a preprint, it's extra important to look at the evidence and not just accept the conclusions. Our analysis is easy to repeat. Please do the same for any criticisms - look for the substance, call out handwaving and appeals to authority.

Link: osf.io/preprints/meta…

I'll put the figures here for ease of reference, but please do read the whole thing (including the Appendix!).

Here's Fig. 1, a timeline of the study. Note that major changes were made 95% of the way through the study period (red & black line), which determined the outcome:

The study was a noninferiority trial. That can make the terminology a bit complex. That's discussed in the main text, but Fig. 2 shows some examples of how altering how you count COVID cases (as happened in this trial) can impact the results you get.

Fig. 3 shows how the successive alterations to the trial changed the outcome. The hypothesis of noninferiority envisages a distribution centred on the black line, and the reported finding of noninferiority required that the 95% CI (illustrated) not touch the red line.

Fig. 4 shows that even minor changes to the reported results would be enough to reverse the reported outcome of noninferiority against the prespecified criterion (although the prespecified analyses were not reported).

Fig. 5 shows how the amount of potential harm that the study was willing to accept as "noninferior" (on behalf of both participants, and those working under policies based on the results of this study) was expanded from the already high value that was initially specified.

Fig. 6 is particularly concerning - there was a statistically significant correlation between male recorded sex and allocation to the more protective, standards-compliant N95 arm. In addition to iniquity of risk, this may bias self-reporting of COVID (the primary outcome).

There were a number of retroactive changes made to the trial registry post publication, which defeats the purpose of registering a trial. Fig. 7 shows an example with additional back-dating that was not recorded in an otherwise nearly-identical protocol dated 2022.05.01.

Fig. 8 compiles a list of critical questions for which answers are urgently needed (CAN/CSA-Z94.4-18 is the applicable Canadian standard for respiratory protection against bioaerosol hazards in the workplace, which specifically includes healthcare workers).

And here are our conclusions. The shortcomings of the study are so significant that they reach beyond it and into any and all processes, policies and materials that made use of the findings.

I would like to thank co-authors @sameo416, @JuliaMWrightDal, @jmcrookston, @GosiaGasperoPhD, @DFisman, and Corinna Nielson (who is also our SPOR Community Advocate), and also the broader community that has surfaced several of these concerns over the past year.

This is a tough topic. Supportive members of three different Faculties have advised that I risk retaliation here in Calgary just for asking difficult questions. But, if academics won't speak up, how can we justify tenure?

This is where the rubber meets the road.

scientificamerican.com/article/masks-…