Thread by @leightjessica

So much of what we hear around RCTs are exciting stories about how evidence is used to inform policy. Which is awesome! I love evidence-informed policy. However, I'm sure many of us have also had experiences that are different, and more challenging.

In the spirit of transparency, wanted to share some different (anonymized) stories about use of evidence. Short 🧵

#1: program has mixed effects (largely null for downstream outcomes). Funding for the implementer concludes, implementer and funder move on. What happens? Brief discussion, draft paper shared (0 replies), almost no policy learning (hopefully research community benefits).

#2: null effects. Findings are rejected. Other researchers engaged as consultants to rebut our findings. Despite ex ante agreement on evaluation design and outcomes, endless discussions are held about how the evaluation was wrong;

was not designed to measure the right effects; could never have in any reality possibly measured the hypothetical very important outcomes the program definitely affected; etc.

#3: intervention has meaningful effects, findings disseminated . Awesome, right? It is, but interestingly a lot of the discussion focuses defensively on outcomes where there wasn’t an impact.

Not sure what the take-away is here. I sympathize: no one likes to hear bad news. That being said, for researchers these discussions can be discouraging. FWIW, these included partners and funders that have an active engagement in research (not evaluation “newbies”).

Another puzzling (related) trend I’ve noticed is the emphasis on qual research as a “insurance policy”: i.e., add a qual substudy so we'll be sure to have some good news. I’m not a qual researcher, but fairly sure that doesn’t make sense.

My fear is that while we hope there are reputational benefits for participating in research for partners, in any particular project those benefits can be uncertain, and the risks of publicizing discouraging program effects (with implications for funding etc.) dominate.

If so, the implications for selection into who participates in evaluations and what is therefore learned are substantial. If it’s going to be easier to share disappointing results, perhaps the reputational benefits of just conducting high-quality research need to grow, a lot.

#EconTwitter