Thread by @the_megabase

genuinely shocking to me that study after study shows that social scientists are no better than regular people at high-level social science predictions: whether studies will replicate, which "nudge" interventions work, how social attitudes will change

a thread of examples:

(to be clear, we're not even talking about superforecasters here, just regular joes)

social scientists, applied and academic, were no better - if anything, possibly worse - than regular people at predicting which interventions would increase gym visits:

https://x.com/SpencrGreenberg/status/1614012813848051712

social scientists *were* substantially worse than regular people at predicting which interventions would make people more likely to get a covid vaccine:

(see pic, which also includes a rather rosy description of the next study in this thread)

https://x.com/katy_milkman/status/1488860537727250432

regular people had larger absolute error (as is often the case) when predicting the effect of incentives on performance of a boring task, but their rankings of the interventions were as good as social scientists':

journals.uchicago.edu/doi/abs/10.108…

social scientists were no more accurate than regular people at predicting or *retrospectively assessing* the social consequences of the covid-19 pandemic:

https://x.com/MattGrossmann/status/1361441337225134081

in a follow-up paper to the one above, social scientists were usually no better than regular people at predicting how general social attitudes would change over time:

https://x.com/AMRotella/status/1623716419941670912

regular people are about as good at social scientists at predicting which studies will replicate:

https://x.com/SteveStuWill/status/1297756278467436545

in estimating gender bias in hiring over time, regular joes' predictions were more extreme than academics' - again, typical - but correlations with real values were very similar, and both made the same mistakes, e.g. thinking there is still bias vs women:

https://x.com/SteveStuWill/status/1723099310898991428

(not grinding a political axe by mentioning the above, that's just what the study says!)

for fairness, here are some papers concluding that social scientists are better than laypeople at various prediction tasks:

1) long-term effects of rct interventions - note studies were mostly in africa (and afghanistan) with predictors in the west:

2) which interventions make people less likely to click "anti-democratic" options in online surveys (...) davidrhysbernard.wordpress.com/wp-content/upl…
pnas.org/doi/10.1073/pn…

3) mask-wearing nudges on democrats and republicans. but - i plotted the results, and it seems most laypeople didn't fully understand the (poorly worded) prediction question and mostly assessed general dem vs rep mask attitudes, not intervention effects.. nature.com/articles/s4159…

a recurring result in these is that expertise level (undergrad vs professor) doesn't matter for accuracy, expert field (economics vs marketing) barely matters - training might make you better at publishing papers in a field, but not (clearly) better at giving actionable advice

a blunt analogy: if a bunch of people went around getting "metereology" phds, and calling themselves "metereologists", but it turned out they were no better than random people at predicting if it would rain tomorrow, i would be annoyed. this is how i see a lot of social science.

@norvid_studies @odoreida mendelian randomization is a kind of iv design used in genetics/epidemiology/medicine in particular and is often used very badly:

https://x.com/Apoorva__Lal/status/1803673184207528159

@norvid_studies @odoreida i have seen some better-informed grumbling about regression discontinuities but my main bone with them is that they often just look a bit absurd. and too easy to find significant differences in random walks:

https://x.com/Scientific_Bird/status/1269735233638871046