Thread by @cbokhove

Let me cite some issues from this paper....

(I know some will keep on insisting that it 'at least is better than not having it all' but I would argue this really depends on what you're looking at. Often it's a trade-off with other things.)

https://twitter.com/cbokhove/status/1307606782295117824

Of course the paper is medicine oriented but given that some like to make that comparison any way... In social science there often are even more challenging limitations. But the 'randomisation' points here also apply....

initial sample selection bias

You really need to check if that doesn't influence outcomes. Random sample in an independent school? Need to check if generalisable more widely. I had that challenge with some Mental Rotation work in an independent school.

achieving-good-randomisation assumption/incomplete baseline data limitation

Although some assume 'random' is enough to say (often with little info), you do really need to check if there really is balance, especially on key traits.

The article also talks about blinding, a tricky aspect in education research. Especially classroom trials virtually impossible to do. But that does remain a limitation then...

Teacher in a school and you run a study and every student knows? That could be an issue. Even more so if you lead an intervention. My point not that useless but that there are limitations and hard to say one per se 'worse' than others...

I have seen RCTs with poor materials that seemed less useful than quasi-experiments with great materials. I have seen qualitative observational studies that gave less insight than RCTs-with-process-eval. Horses for courses.