megabase Profile picture
Jul 12 15 tweets 6 min read Read on X
genuinely shocking to me that study after study shows that social scientists are no better than regular people at high-level social science predictions: whether studies will replicate, which "nudge" interventions work, how social attitudes will change

a thread of examples:
(to be clear, we're not even talking about superforecasters here, just regular joes)

social scientists, applied and academic, were no better - if anything, possibly worse - than regular people at predicting which interventions would increase gym visits:
social scientists *were* substantially worse than regular people at predicting which interventions would make people more likely to get a covid vaccine:

(see pic, which also includes a rather rosy description of the next study in this thread)

Image
regular people had larger absolute error (as is often the case) when predicting the effect of incentives on performance of a boring task, but their rankings of the interventions were as good as social scientists':

journals.uchicago.edu/doi/abs/10.108…
Image
social scientists were no more accurate than regular people at predicting or *retrospectively assessing* the social consequences of the covid-19 pandemic:
in a follow-up paper to the one above, social scientists were usually no better than regular people at predicting how general social attitudes would change over time:
regular people are about as good at social scientists at predicting which studies will replicate:

Image
in estimating gender bias in hiring over time, regular joes' predictions were more extreme than academics' - again, typical - but correlations with real values were very similar, and both made the same mistakes, e.g. thinking there is still bias vs women:

Image
(not grinding a political axe by mentioning the above, that's just what the study says!)

for fairness, here are some papers concluding that social scientists are better than laypeople at various prediction tasks:
1) long-term effects of rct interventions - note studies were mostly in africa (and afghanistan) with predictors in the west:

2) which interventions make people less likely to click "anti-democratic" options in online surveys (...) davidrhysbernard.wordpress.com/wp-content/upl…
pnas.org/doi/10.1073/pn…
3) mask-wearing nudges on democrats and republicans. but - i plotted the results, and it seems most laypeople didn't fully understand the (poorly worded) prediction question and mostly assessed general dem vs rep mask attitudes, not intervention effects.. nature.com/articles/s4159…

Image
Image
a recurring result in these is that expertise level (undergrad vs professor) doesn't matter for accuracy, expert field (economics vs marketing) barely matters - training might make you better at publishing papers in a field, but not (clearly) better at giving actionable advice
a blunt analogy: if a bunch of people went around getting "metereology" phds, and calling themselves "metereologists", but it turned out they were no better than random people at predicting if it would rain tomorrow, i would be annoyed. this is how i see a lot of social science.
@norvid_studies @odoreida mendelian randomization is a kind of iv design used in genetics/epidemiology/medicine in particular and is often used very badly:
@norvid_studies @odoreida i have seen some better-informed grumbling about regression discontinuities but my main bone with them is that they often just look a bit absurd. and too easy to find significant differences in random walks:

• • •

Missing some Tweet in this thread? You can try to force a refresh
 

Keep Current with megabase

megabase Profile picture

Stay in touch and get notified when new unrolls are available from this author!

Read all threads

This Thread may be Removed Anytime!

PDF

Twitter may remove this content at anytime! Save it as PDF for later use!

Try unrolling a thread yourself!

how to unroll video
  1. Follow @ThreadReaderApp to mention us!

  2. From a Twitter thread mention us with a keyword "unroll"
@threadreaderapp unroll

Practice here first or read more on our help page!

Did Thread Reader help you today?

Support us! We are indie developers!


This site is made by just two indie developers on a laptop doing marketing, support and development! Read more about the story.

Become a Premium Member ($3/month or $30/year) and get exclusive features!

Become Premium

Don't want to be a Premium member but still want to support us?

Make a small donation by buying us coffee ($5) or help with server cost ($10)

Donate via Paypal

Or Donate anonymously using crypto!

Ethereum

0xfe58350B80634f60Fa6Dc149a72b4DFbc17D341E copy

Bitcoin

3ATGMxNzCUFzxpMCHL5sWSt4DVtS8UqXpi copy

Thank you for your support!

Follow Us!

:(