Thread by @cbokhove

There are loads of things that matter in good research. There is an assumption that if one of the ‘gold standards’ criteria isn’t met, it can’t be good research. I would rather say that it just has a limitation. It would not be good to think b/w here.

What some also seem to forget is that all those criteria matter. So, it’s great you randomised participants but if your measurement is bad....it’s still bad. Or if your comparison groups are poorly chosen....still poor.

Or take intervention materials. You can get everything ‘right’, but if your materials are poor and unlikely to ever be used in a classroom (understandable, maybe you are trying to ‘control’ other things and kept it simple), can we then rely on the findings?

I have seen exemplary RCTs but if you then drilled down to the sample, there were obvious imbalances. Sometimes ‘social science’ or ‘education reseach’ is criticised (pointing towards medicine for example) but it can pose issues there as well. tandfonline.com/doi/full/10.10…

Some find all these tweets perhaps ‘postmodern’, but you’d be mistaken to think that. I value rigorous design, I value rigorous measurements. Whether qualitative or quantitative, we need to do the best research we can. But caricatures are pointless.

Perhaps even working complementary is best, possibly in teams.
An observation study of teachers.
Some secondary data analysis.
A focus group with students.
Some design reseach to improve the materials.
A trial of the materials in a large randomised sample.
Etc.