Thread by @DFID_RED_GCSD

Thread <RANDOMISTA HARVEST>

It’s been a few weeks since we were last called the R-word but it set us thinking about a summer holiday thread.
Plse forgive the mix of B-movie pulp & serious research. But if it helps to retire <Randomista> it was worth it🤨
peter, gcsd
1/21

🤨<low quality pastiche begins>

RANDOMISTA!
That cry is rarer now. In the worst times, any research conference could be disrupted by shouted denunciation. Or by rumours and unease - Randomistas had been spotted in the lobby, with offers of ‘gold standard’ evidence.
2/21

🙄...Folks said the Randomistas were quick & effective. Or was it efficacious? One moment you were fan-tweeting how Participatory Rural Appraisal changed your life - the next you were Twitter trashing a whole field because of one nil effect RCT. You’d been harvested.
3/21

😟But I remain vigilant.

If 'Randomista!' rings round a spiral-carpeted conference plenary, I start my defensive drills. I mouth the litany of research methods that I know and love, and thank each in turn for sparking research joy...
4/21

😰...then I run through the RCTs we fund, total the budgets, & calculate the % of our entire annual research budget that is spent on RCTs. RCT spend has hovered around 15% per year for the last 5 years. I check again - still ~15%.

We are <still> not Randomistas.
5/21

<<But seriously>>
This 15% of our annual budget on RCTs (~2 of 18 prog) supports some brilliant work – as does the 85% spent on 16 progs that use a wide range of mixed method, inter-disciplinary, politically-informed, qual ‘n’ quant research methods.
6/21

Our biggest RCT investment is with @JPAL & @poverty_action -
- Governance Initiative;
- Crime & Violence Initiative;
- Peace & Recovery Initiative.
Some JPAL & IPA RCTs have really shaken my assumptions about how things work - in key governance areas.
3 examples:
7/21

Eg1: I had struggled with police reform projects, with experts promoting pet models for training & police station reform, but little sense of what success would look like or what to measure.
This Rajasthan policing RCT helped cut through my fog bit.ly/2YtowQm
8/21

Eg2: Post Conflict Community Reconciliation – surely only good, no?
This Sierra Leone RCT found +ve community level benefits but at expense of -ve mental health effects (re-living trauma) for individuals. bit.ly/2p2Mj7e
9/21

Eg3: It’s well known that for community health workers, locals do better...
This Zambia RCT tested recruitment methods: community motivation vs advertising career opportunities.
Careerists performed better, improved health practices & outcomes. bit.ly/2SQ5tdR
10/21

So was I harvested?

No. These were in places where I was NOT working. No single study Eureka! moment caused a rash U-turn. But the insights were a 1st challenge to longheld assumptions about critical points in complex systems, with big $ resting on these assumptions.
11/21

Thoughts also count:

Discussing doing a RCT doesn’t always end in a RCT. But I have found ‘dry runs’ help thinking about theory of change; interventions (min. # of community meetings needed to deliver change?) what success looks like (outcomes); how/when to measure.
12/21

And let’s hear it for the evaluated:
Huge respect to practitioners with emotional/ financial/ professional/ social investment in an intervention who volunteer for evaluation. And not just because donors want it…. @INBreakthrough are among the boldest I know
13/21

Our 2nd major prog is @ICED_LSHTM 's new £7m PENDA Prog for Evidence to Inform Disability Action ~10 Impact Evals incl RCTs. One reason for including RCTs was to push ambition in ‘what works?’ outcome measurement – do interventions change lives? bit.ly/2MlV88l
14/21

RCTs also emerge in our wider portfolio when the best means of addressing a RQ is an RCT– e.g. on tax communication with @ICTD & Rwanda Revenue Authority. Non-randomistas have delivered great RCTs & safely returned unscathed to normal life...
bit.ly/2YrAmGB
15/21

Kicking RCTs:
We find that RCTs attract far more criticism than any other type of research in our portfolio. Typical criticism:

a. Ethics: usually about withholding [untested!] treatment from control group. We find RCT researchers are always happy to discuss this.
16/21

[...whisper... Sometimes if practitioners question an RCT's ethical approach, we describe the paper trail of ethical consideration & approval; then we ask about ethical oversight of their own programmes and activities.

Sometimes awkward silence is the reply...].
17/21

Healthy kicks (continued...)
b. an RCT’s findings (and claims of demonstrating causality);
c. artificiality of breaking complex programmes into chunks that can be tested in isolation.
d. Generalisability.

18/21

Tyre kicking for all!

These challenges are healthy – but tend to target RCTs alone. I think that all research can benefit from healthy 'tyre kicking'. The 4 ‘kicks’ above are relevant for lots of research, not just RCTs.
e.g. heroic un-evidenced causal claims...

19/21

RQ Takes All:
We believe that ‘the method chosen should be the best for answering that RQ’. We add “within available resources e.g. funds, skills, time, risk”. Where do RCTs fit in wider bodies of evidence? See neat precis by @Howardnwhite bit.ly/2LM0v13
20/21

Randomista Harvest: Epilogue

Old man sits on porch, strokes genetically modified bee, talking to 2 holograms.

Grandpa! In history we’re studying the 'Randomistas'. Which side were you on?

"Both sides kids. Both sides*"

(*Ph 1 control then treated in Ph 2 roll out).
21/21

Personal views, not policy! Peter Evans, GCSD.
If you are having a holiday - have a good one.