Thread by @leightjessica

1/ Wanted to do an #EconTwitter 🧵 on a new + important topic that's growing in the literature: rigorous evidence about how policy-makers use + respond to evidence! Most of these papers are very recent, many still WP

2/ One published in AER 2021 by @HjortJ @dianamoreira_sb Rao and Santini; an experiment w.mayors of 2,150 Brazilian municipalities; they find mayors are WTP for evidence, and update priors upon receipt; value large samples more, but not dev country studies
aeaweb.org/articles?id=10…

3/ Relatedly, they show that mayors briefed on the effectiveness of one policy (tax reminder letters) are 10 pp more likely to adopt it

4/ Recent WP by @mrsultan713 Naseer and @Daniel_L_Chen analyzed the effect of econometrics training for Pakistani policymakers, also recently on DevImpact blog
users.nber.org/~dlchen/papers…
blogs.worldbank.org/impactevaluati…

5/ Imp, these are higher-level policymakers (not bureaucrats) - deputy ministers. Exposure to a serious econometrics workshop ⬆ perceived imp of quantitative evidence, ⬆ WTP for RCT evidence and ⬇ WTP for correlational evidence. Effects are large + nicely captured in graphs

6/ Shifts in broader attitudes toward causation

7/ Shifts in priors around effects of deworming after exposure to evidence

8/ Shifts in WTP for RCTs

9/ Also relevant the very interesting work of @evavivalt Coville @sampskc who ran experiments at WB and IADB to understand how policymakers + practitioners weigh evidence. evavivalt.com/wp-content/upl…

10/ Found that that they place substantial wt on characteristics linked to external validity (local experts, context in which evidence collected); while researchers place more emphasis on internal validity.

11/ Assessing WTP in terms of impact, policymakers state they'd accept a program with 6 pp lower impact, if it's been recommended by a local expert. (A big gap - many programs have effects smaller than this)

12/ Related paper by same team on how researchers, policymakers + practitioners update beliefs shows that generally policymakers are more optimistic; but, when presented w/IE findings, all 3 groups update asymmetrically (focusing on good news)
evavivalt.com/wp-content/upl…

13/ Another published paper by @Sheheryarbanuri Dercon and @varungauri conducted experiments w/policymakers at the WB and DFID. Found some evidence of ideological interpretation of evidence:
academic.oup.com/wber/article-a…

14/ Policymakers less accurate in interpreting findings presented wrt to min wage, as opposed to neutral framing (the effect of face cream). Also, susceptible to framing: more likely to choose risky option when framed as losses. (Deliberation mitigates the 1st of these biases)

15/ These are all dev-oriented papers; there are certainly US-focused papers (including on education specifically) that I have missed, as well as the recent paper by @mattietoma analyzing responsiveness of US policymakers to program estimates, thread here

https://twitter.com/leightjessica/status/1501649545490481164

16/ Leaving those for others to add; or for another 🧵 ! Will conclude with the (un-original) thought: the body of research here is much smaller than the importance of the Q: surely slightly ⬆ uptake of existing evidence, at least in some domains, could have very large effects

17/ Effects could be larger, in fact, than ⬆ our body of knowledge (which we already invest in - and which I endorse!); particularly if we understand more about how to ⬆ evidence uptake in large countries, + those w/large # of poor people. Clearly much more for us to learn.