Thread by @Josh_Merfeld

New report just dropped. Time for a thread.

You can find the replication report and the author's response in the linked thread (in the second tweet).

https://twitter.com/I4Replication/status/1899508739117391910

Let me start with a small point: I hate the title of the replication. It refers to a comment in a do-file but, to be perfectly honest, it's entirely possible that it was just poor word choice from an RA whose first language is not English. The data is much more damning.

First up: there again appears to be overlap with other papers that have "different" treatments! As with earlier replications, this overlap is not discussed in any of the three papers at issue here.

Second: paper says that 76 primary schools were randomly assigned to treatment or control. But here's the thing: they were very clearly NOT randomly assigned. There are nine unions (admin areas) in the sample. Five are entirely treatment and four entirely control.

This is essentially impossible with actual randomization. Author claims "the contractor, using his practical knowledge, grouped treatment schools within contiguous unions to address local concerns". Since "balance" checks are okay, author claims internal validity is also okay.

But that is NOT how internal validity with RCTs works. In addition, not catching this "error" would be indicative of amazingly poor RCT management and data management. He also had the contractor randomly assign schools. What? No researcher I know does that.

Strike one.

In the author's response he names the contractor and blames him for everything. This is just honestly poor behavior. No shame.

There is clear inconsistency in the description of the data collection process and the actual data. It's hard to know exactly what's going on because the documentation isn't clear, so I'll move on from that point.

In the data itself, there is a mismatch in baseline test scores for the SAME STUDENTS in different datasets. Data was collected more than a decade ago, so some of this could be manual entry error, but there are a LOT of mismatches. Seems sus.

The really damning thing here is that there is systematic differences in this mismatch between the control and treatment groups. Treatment seems okay, but the control group's baseline values across datasets are different. Looks like control's baseline values were adjusted up!

Note that this matters because higher baseline in control means smaller increase at endline...

There's a huge amount of abnormal bunching in the test scores, and that bunching is MUCH worse in the control group. (I'd also note that the baseline and endline distributions are wildly different? Best-practice RCT management would be to use the same instrument in both.)

I mean what even is this? There is no way these data came from the same instrument. It's impossible.

The author did a household survey to ask certain questions, saying that households were randomly sampled from the treatment and control groups. But there is a HUGE disparity in the probability of having in survey across the groups for one of the two grades only.

Skipping forward a bit... different schools have different treatment statuses in different analyses. This alone could be sloppy data work or bad data entry, but it's just another "what?" in a long line of whats.

I find this particularly problematic: the author said that the household survey asked about parents' opinion re: the intervention (parent-teacher meetings). But: control parents were more positive about the intervention? Very very strange.

As far as I can tell, the author's response just says that these opinions are a secondary outcome, so no big deal. I don't think that's the right way to tackle this if the data are real.

Another thing that is very concerning: If the intervention was done as described, parents had to spend 15 minutes with parents of each child. Class size is around 59 on average, meaning that's an additional 15 hours per month per teacher.

But teachers were paid only two dollars more PER YEAR for the intervention. Author says "the intervention [was] remarkably low cost," but that seems misleading, at best.

Replicators do some additional robustness checks. Probably won't be surprised to learn that the results don't hold up.

TLDR: I don't believe a word in the original study.