Thread by @WesPegden

Quick thread about re-randomization in randomized trials, illustrating the following two simple points:
*) re-randomization for balance is sometimes fine and doesn't affect significance thresholds
*) re-randomization for balance can sometimes cause big artifacts.

1/14

First scenario, consider a clinical trial, where we randomize individuals independently (by individual coin flip, say) to treatment or control groups.

What if we re-randomize until the two groups have exactly the same size?
2/14

This is the same as choosing an equal split uniformly randomly, and standard statistical approaches will still apply without problems.

It is similar if one re-randomized until the populations were within some threshold, or if this was true for subgroups (sex, race, etc).
3/14

Re-randomization can be reformulated as conditioning on a certain event (the condition of acceptance), and in these particular cases above, the corresponding event gives rise to a conditional probability space with well-understood and definable structure.
4/14

In the case of cluster trials, I might imagine re-randomizing to balance the number of "big" of "small" villages in each trial arm. This would be equivalent to the similar examples above.

It tempting, but incorrect, to think that balancing summary statistics is always ok too.
5/

Example: Suppose I conduct a cluster randomized trial in 20 villages, whose populations are
1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2,11
(nine have pop 1, ten have pop 2, and one has pop 11).

If I define a notion of a "big village" and re-randomize to balance big villages: 👍
6/14

If I run my trial and find all the treatment villages do better than all the control villages, this will be a highly significant finding at something like p=.000001, and this subgroup balancing will not have a big affect on this.

7/14

Suppose now, instead, that I choose a random 10:10 split of villages but re-randomize to balance the mean or total population of the villages in the two groups, rather than balance in predefined subgroups like "big" or "small". Seems okay, right?

8/14

But this situation is a disaster.

If I rerandomize a *lot* of times, I will essentially always find one of two splits:
11,1,1,1,1,1,1,1,1,1 vs 2,2,2,2,2,2,2,2,2,2, or
2,2,2,2,2,2,2,2,2,2 vs 11,1,1,1,1,1,1,1,1,1

In particular, all the randomness in my trial is now 1 coin flip.9/

If, for example, I am studying infectious disease, which all of the villages with >1 person have and none of the villages with 1 person have, I have a 50% chance of finding 9/10 of my treatment villages healthy while 10/10 of the controls are sick.

10/14

With standard tests this would appear to be extremely significant at something like p=.0001 but really it is a 50% chance event.

To conclude anything better, I have to make tacit assumptions about the generative distribution of village sizes, or its effect on my study outcome.

Even if I don't randomize a lot of times, it is still true that the more times I randomize, the more pop-1 villages's will cluster with each other (and the 11 village) and, conversely, the more pop-2 villages will cluster together.

12/14

If I randomize 100 times, my split will essentially be a 1-in-100 outlier with respect to how many pop-1 villages cluster on one arm. In the scenario above, where villages with only 1 person all are free of disease, I can easily get a p=.01 outcome just from the rerandomization.

All this is to say that while re-randomization can be fine and have advantages, it is worth remembering that it is not universally valid for all conditions we might intuitively think of as "balance", and one should think carefully when employing it at the outset of a trial.
/14