Thread by @leightjessica

1/ Enjoyed seminar by @Susan_Athey at Georgetown yesterday presenting paper about the effects of contraceptive counseling + discounts in Cameroon, + an overview of process of running an adaptive RCT.
Short #EconTwitter 🧵 about the latter, for interested applied researchers

2/ (Not touching on the paper + findings itself - fascinating, but already very well-covered in the blogs by @BerkOzler12)
blogs.worldbank.org/impactevaluati…
blogs.worldbank.org/impactevaluati…

3/ High-level points: goal of an adaptive RCT is to automate process of refinement (run trial comparing multiple treatments to control; identify the best one; test it further; etc.) Designed to replace human time w/computing time as it runs

4/ But, extensive human time needed in planning (esp the 1st time!) In the planning phase, imp to identify: the length of the study; the data collection algorithm; the objective or loss function (what will the bandit algorithm optimize).

5/ Also important to think about the high-level goal of the trial: maximize benefits for a particular group? Conduct a particular test of policy or policies? (Normally, in our field trials, both goals are important, + we seek to balance both)

6/ The first phase (pilot phase) of the RCT itself will be a simple RCT, comparing treatments to control; but, over time, the bandit algorithm will gradually measure the effectiveness of treatment arms + assign more sample to better-performing arms (less to worse-performing)

7/ This will ⬇ the error bands around the former estimate (better-performing), but ⬆ them for the latter. A tempered Thompson algorithm will balance the goal of max welfare for those in the study, w/the goal of learning more about optimal policies.

8/ And then in the final “evaluation” phase, participants will be assigned to the learned “optimal” policy with a higher probability, in order to compare this to the control arm.

9/ This was a fascinating talk and made a compelling case for considering these trial designs, though clearly considerable methodological inv required the 1st time to do it well.

10/ Some more links: here is an interesting paper by @caria_stefano Gordon @maxkasy @simonrquinn Shami @t8el reporting on an adaptive experiment analyzing job search assistance for refugees
conference.nber.org/confer/2021/YS…

11/ (This, along w/Cameroon trial, are one of a few - or the only? - dev-country adaptive RCTs that have been completed, afaik.) Also an overview of adaptive experiments forthcoming in ECMA by Kasy and @ASautmann here
maxkasy.github.io/home/files/pap…

12/ As well as a dashboard to play with (using prev outcome data) here - something I look forward to doing.
maxkasy.shinyapps.io/exploration_sa…