Thread by @leightjessica

1/ Was excited to see #EconTwitter interest in evidence on how policymakers use evidence

https://twitter.com/leightjessica/status/1503800956714160132

Follow-up 🧵. First as usual, I aim to be interdisciplinary, but can't be comprehensive; a bias toward econ in these threads. Adding links + cites always welcome.

2/ Second, bc I'm focusing mostly on econ, economists have a comp advantage in analyzing how policymakers use econ (as opposed to other types of knowledge). Hopefully other disciplines are working in parallel - we should def understand how policymakers work w/other evidence.

3/ Free research idea: an interdisciplinary team should run a study analyzing how policymakers respond to multiple forms of knowledge and analyze it interdisciplinarily. Would love to work on this myself!

4/ That being said, goal for today was to highlight a bit more of the US or rich-country evidence about policymakers using evidence. First. h/t @ForrestFleisch1 for pointing out imp foundations of this literature from Weiss
jstor.org/stable/3109916…
jstor.org/stable/42783234

5/ Key argument here: focusing on policymakers' specific use of a research finding in a given decision is only a small part of the overall impact of research on policy. In which case imp for researchers to consider how to capture these effects (quantitatively or otherwise)

6/ Back to more recent econ papers focusing on policymaker use of evidence. Thanks to several (h/t @SarahCohodes) who flagged @nozominaka JMP analyzing US educ policymakers
nozominakajima.github.io/files/nakajima…

7/ Show that educ policymakers prefer studies that are large, multisite, + similar to own jurisdictions; but, have no strong preferences over research design. They also update their priors when presented w/research evidence; only if it's presented accessibly

8/ Another education-related paper by @peterbergman_ Lasky-Fink and @Todd_Rogers_ showed that policymakers have meaningfully wrong priors about 1 Q (effects of enrollment defaults); but, update + are willing to pay more when info is provided
studentsocialsupport.org/files/s3rd/fil…

9/ @thaddunning @guygrossman @maqartan @dshyde McIntosh @GarethNellis also present evidence that policymakers are more responsive to evidence from RCTs if they are multisite, part of @EGAPTweets project
google.com/books/edition/…

10/ Moving over to health, there's a much larger literature around how policy is used in health (much of it qualitative or observational); incl a number of systematic reviews, (third is EU only)
link.springer.com/article/10.118…
journals.plos.org/plosone/articl…
sciencedirect.com/science/articl…

11/ I'll conclude w/2 papers that are really dev-related papers omitted in error from my earlier thread. @leecrawfurd @JustinSandefur @susannahhares Minardi conducted survey experiments w/900 policymakers in 35 countries (LMICs)
cgdev.org/publication/un…

12/ In addition to providing interesting stylized facts about preferences, also show how policymakers respond to evidence provided. Similar to other studies, policymakers are very responsive to context + sample size, but very unresponsive to variation in study design

13/ Final paper by Rogger and Somani, working with a large sample of bureaucrats in Ethiopia, finds that avg rate of errors around basic admin facts is high; but much higher when officials have less ind authority (b/c, in essence, they have weak incentives to acquire info)

14/ An information provision intervention is effective in reducing errors, and those effects concentrated in these low-management-quality orgs. (But note this is just "information" as in facts, not evidence from policy evaluations.)

15/ To conclude again, w/some simple thoughts: in general, policymakers across a range of contexts value scale of studies (sample size + multisite) + proximity (same or similar contexts) seemingly much more than researchers (who usually value pubs, w/huge premium on originality)

16/ The policymaker objective function would suggest we should be doing many more multi-site trials + evaluation replications of similar programs in local sites (perhaps not necessarily RCTs, but other high-quality quasi-experimental designs) than we are right now. End 🧵