Thread by @JustinSandefur

🚨 New RCT results 🚨

"Beyond Short-term Learning Gains: The Impact of Outsourcing Schools in Liberia after Three Years" -- by @marome1 and me.

Working paper:
cgdev.org/sites/default/…
Policy brief:
poverty-action.org/sites/default/…
Thread:
1/

@marome1 The basic question here is whether poor countries, with limited state capacity, can improve public services by outsourcing delivery to the private sector.

2/

@marome1 Quick backstory: This all began with Liberia’s plan to outsource *all* its public primary schools to a for-profit chain of private schools run by an American couple, @BridgeIntlAcads.

3/

After pushback from teachers unions, civil society, & aid donors, the Liberian govt rolled back that plan to a more modest pilot, launched in 2016, with a bit more competition and oversight:

a) 93 schools, not 2000+
b) 8 different operators, not 1
c) A randomized evaluation

4/

Of the 93 treatment schools in the RCT, @BRACworld ran 20, @BridgeIntlAcads 23, @Omega_Schools 19, @risingacademies 5, @streetchilduk 12, @MovementYmca 4, @MoreThanMeORG (now known as Hilltop Schools) 6, and Stella Maris (who basically never showed up to their schools) 4.

5/

@BRACworld @BridgeIntlAcads @Omega_Schools @risingacademies @streetchilduk @MovementYmca After the first year, we found in our previous paper (forthcoming in the AER w/ @wsandholtz) that outsourcing led to modest learning gains, but at very high cost, and with some unintended consequences...
aeaweb.org/articles?id=10…

6/

@BRACworld @BridgeIntlAcads @Omega_Schools @risingacademies @streetchilduk @MovementYmca @wsandholtz One unanticipated outcome was Bridge expelling kids en masse from some of the schools they took over -- as reported in the press, and corroborated in the RCT data.

7/

@BRACworld @BridgeIntlAcads @Omega_Schools @risingacademies @streetchilduk @MovementYmca @wsandholtz Much worse: a ProPublica investigation revealed the shocking extent of sexual abuse in schools run by 1 of the 8 operators, "More than Me". While these crimes occurred prior to the start of the outsourcing program, the cover-up extended well into it.
propublica.org/series/unprote…

8/

@BRACworld @BridgeIntlAcads @Omega_Schools @risingacademies @streetchilduk @MovementYmca @wsandholtz The hope was that over time safeguards had improved, test scores would rise, and costs would fall...

Which brings us back to our new follow-up paper. What did we find after 3 years?

9/

@BRACworld @BridgeIntlAcads @Omega_Schools @risingacademies @streetchilduk @MovementYmca @wsandholtz First, beyond year 1, learning gains plateaued. Outsourcing raised English scores by 0.18sd after 1 year, and after 3 years… by 0.16sd. That’s about 4 extra words per minute in reading fluency, i.e., not huge.

10/

Second, beyond learning gains, outsourcing had mixed effects. Corporal punishment (though still widespread) went down, but self-reported sexual abuse didn’t. And notably, dropout increased significantly — driven by Bridge-run schools, and girls who reported pregnancy.

11/

Third, looking beyond averages, some orgs posted null results, or stark trade-offs between test scores and other goals. But others like @streetchilduk posted uniformly positive results without major unintended consequences, and adhered to the $50/pupil target.

12/

@streetchilduk In sum, we find mixed impacts of outsourcing schools in Liberia. Narrow fixation on test scores came with trouble in other domains. But uniform contracts produced very non-uniform results across operators, positive and negative.

We hope funders take note of that variation.

13/

@streetchilduk P.S. This is a rare RCT run over the objections of some orgs being evaluated. Privately, some of those orgs have tried to block our funding, get us fired, etc. Publicly they have spun our past results as an unqualified success.

Read and make your own judgments.

14/

For full transparency, we've posted replication data & code for year 1 on dataverse
dataverse.harvard.edu/dataset.xhtml?…
We’ve submitted year 3 replication materials to IPA’s data team; it should be public as soon as they confirm all personally identifiable information has been removed.

15/

Finally, RCTs involve 100s of ppl. @GeorgeKWerner & @Fihi_maFihi insisted on an evaluation. The @poverty_action team led by @GDackdolo, @ArjaDayal & @OsmSiddiqi ran it. @UBSoptimus @ESRC and Aestus kindly funded. And @RRMaximiliano wrote much of the year 3 Stata/R code!

/end