,
7 tweets,
2 min read
Read on Twitter
Important new paper from Larry Orr, Rob Olsen, @lizstuartdc, and others. onlinelibrary.wiley.com/doi/abs/10.100…
In thinking about evidence-based policy, policymakers might often wonder, "Will the results of one evaluation/RCT apply to my jurisdiction?"
Orr et al. look at three RCTs in education (one on 36 charter schools, one on ed tech in 132 schools, and one on 84 Head Start centers).
Orr et al. look at three RCTs in education (one on 36 charter schools, one on ed tech in 132 schools, and one on 84 Head Start centers).
They ask, "Could you have predicted the results in a given school/center, by relying on the results from all the other schools/centers in that same RCT?"
Answer: not very well.
Answer: not very well.
Conclusion: "most localities will not be able to use the results of large-scale
multisite evaluations to accurately predict the likely consequences of adopting an intervention or policy."
multisite evaluations to accurately predict the likely consequences of adopting an intervention or policy."
But it gets worse:
First, it would likely be even more difficult to extrapolate to other sites that didn't join the RCT in the first place.
First, it would likely be even more difficult to extrapolate to other sites that didn't join the RCT in the first place.
Second, the analysis doesn't account for how an intervention's effects might change over time.
* On temporal external validity, see nber.org/papers/w22449.
* On temporal external validity, see nber.org/papers/w22449.
This paper seems usefully paired with Lant Pritchett's 2013 work on external validity, which argues that "OLS estimates from the right context are, at present, a better guide to policy than experimental estimates from a different context." cgdev.org/publication/co…
