Share this page!

Wes Pegden Profile picture
Twitter logo
2 Dec, 11 tweets, 4 min read
This blog post discusses our re-analysis of the recently released data from the Bangladesh mask trial (with @ChikinaLab and @beenwrekt).

An accompanying short note is linked from the blog (arXiv link later).

Thread what we find: 1/
We carried out simple analyses of this data using standard nonparametric paired statistical tests.

Through this lens, there are highly significant effects on behavior, but the primary outcome (symptomatic seropositivity) is not significant. 2/..
One of the striking things in the figure above is the imbalance in the size of the consenting populations between the (50:50 randomized) treatment/control groups. There are highly nonrandom differences in the rates consent staff reached households in treatment/control villages.
This cautions against using rates rather than counts of outcomes in the trial, as rate differences are mostly driven by differences in denominators rather than numerators.

The 1086:1106 split in primary outcome counts would not be significant even with individual randomization.
More generally, highly significant effects on behavior (not only on participants with respect to physical distancing, etc, but on how staff interact with participants) suggests subtle differences in bias-susceptible survey-dependent endpoints requires cautious interpretation.5/
Another way of putting this is that staff were 50% more likely to deem people "unreachable" in control than treatment villages.

Fundamentally, big differences even in staff behavior make it difficult to reason about small diff's in survey-based outcomes.

As we discuss in our writeup, the total final difference in size is a full 9%, and arises not just from significant diff's in the step you are discussing here, but also, curiously, from an earlier step where staff were supposed to be blinded.

Notably, this final effect size of bias in population denominators, 9%, is roughly the same size as the effect size found in the RCT paper for the effect of rates. It is large enough to be concerning.

The point of RCT's is to avoid problems of failure of imagination. Mariginal "barely consenting" households might be
*) healthier
*) more distant (socially geographically) and thus at lower risk
*) have different survey behavior
etc
To actually bound the bias, you would do robustness checks to see whether how you handle "unreachable" or unmapped households affects significance for your primary outcomes. What if you 0 them out? What if you assume they test + in endline 1/2 as often?
As a broader point, the one thing we all agree on is that the intervention was successful at having a very real impact on both staff and participant behavior and interaction.

We should agree that reasoning rigorously about subtle survey-based effects is difficult in this context

• • •

Missing some Tweet in this thread? You can try to force a refresh
　

Keep Current with Wes Pegden

Wes Pegden Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

More from @WesPegden

Wes Pegden Profile picture
3 Dec
Quick thread about re-randomization in randomized trials, illustrating the following two simple points:
*) re-randomization for balance is sometimes fine and doesn't affect significance thresholds
*) re-randomization for balance can sometimes cause big artifacts.

1/14
First scenario, consider a clinical trial, where we randomize individuals independently (by individual coin flip, say) to treatment or control groups.

What if we re-randomize until the two groups have exactly the same size?
2/14
This is the same as choosing an equal split uniformly randomly, and standard statistical approaches will still apply without problems.

It is similar if one re-randomized until the populations were within some threshold, or if this was true for subgroups (sex, race, etc).
3/14
Read 14 tweets
Wes Pegden Profile picture
27 Sep
The authors of this study are doubling down on @Jabaluck's "guesstimate" of the # of lives masks would save in the U.S. based on their (impressive, sorely needed) cluster trial in Bangladesh.

There are many reasons this guess is speculative to the point of not being useful.
1/6
Apart from the obvious things about translating a finding which was highly context specific (the benefit concentrated in particular ages and in villages using a particular mask type) to a completely different context, one cannot make these estimates ignoring epidemic dyanmics. 2/
E.g.:
Suppose every K infections results in death.

To a first approximation, to believe "masks for a year" prevents x deaths, you either have to believe that it increases by K*x the number of people who will never get infected, or the number who would get vaccinated first.
3/
Read 6 tweets
Wes Pegden Profile picture
1 Sep
Everyone should look at the remarkable work done in this cluster randomized trial.

They found that an intervention which increased surgical mask uptake in community settings significantly reduced SARS-CoV-2 infection among older adults.

A thread.
1/6
Most people's 1st tendency is to claim studies like this support what they already knew. In reality, the study had specific and not necessarily intuitive findings.

The study even collected predictions from experts, and found that they failed to predict the study outcomes!
2/6
For example, the study found that increasing mask usage had statistically significant effect on SARS-CoV-2 infection.

But these results were driven by surgical mask use, and by reductions in infections in people over 50.

A brief reminder that CDC mask guidelines start at age 2.
Read 9 tweets
Wes Pegden Profile picture
31 Aug
On Friday, epi-Twitter exploded with urgent discussion of an MMWR on an isolated incident in which 1 teacher infected an unusual number of students.

Receiving much less attention was an MMWR on the same day of lower case rates all last year in LA schools than in the community.1/
At the time of this writing, the MMWR with data had 74 retweets and the MMWR on that one time this one crazy thing happened had 1.1K retweets, many from serious people claiming that this report affected in some fundamental way our understanding of COVID-19 risk in schools.
2/
Data is boring and stories seem compelling. But scientists and public health agencies should be actively working against the natural tendency to give greater weight to outlier incidents than data-driven understanding of risks.
3/
Read 4 tweets
Wes Pegden Profile picture
31 Aug
One idea that has not been discussed much is the question of whether regulators have a special role to play in deciding when coercive measures can be used to increase uptake of vaccines.

In practice the hurdle for this has just been EUA, not even full approval.
1/
I think it is worth thinking about what principles should guide the decision of when coercive measures are ethically appropriate and whether regulators should play a role in adjudicating when that bar is crossed.

2/
The current situation is that we allow mandates even in cases where no clinical trial has weighed the direct individual risk/benefit (e.g., mandates for individuals with confirmed previous infection. This may also be the case soon for boosters).

This certainly seems flawed.

3/
Read 4 tweets
Wes Pegden Profile picture
27 Aug
Just a reminder that (unlike for existing vaccination regimens) there are no trials quantifying a clinical benefit from boosters.

It's remarkable to see the decision portrayed here as one being made based on conversations between political leaders.

1/
cnbc.com/2021/08/27/bid…
Needless to say, politicians that have already made the decision to push forward with early boosters have an incentive to sell this decision to others as a wise and prudent one. Indeed, after implementing this decision, trials would not serve a helpful political purpose.

2/
But for the actual people who will receive boosters (let alone those who might have received a 1st dose of vaccine had the dose not been used as a booster in the U.S.) questions about whether boosters actually have any real clinical benefit (vs small risks) are crucial.

3/
Read 4 tweets

Did Thread Reader help you today?

Support us! We are indie developers!

This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Too expensive? Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us on Twitter!

Like this author's thread?

Get emails when @WesPegden has new unrolls!

Check out other threads from @WesPegden!

Read all threads by @WesPegden

:(