Thread by @MatthewAKraft

[THREAD] I've grown increasingly frustrated w/ outdated & unrealistic interpretations of effect sizes that ignore costs & scalability . . . so I decided to propose a new approach

"Interpreting Effect Sizes of Education Interventions"⬇️. I welcome feedback
scholar.harvard.edu/mkraft/publica…

The default approach to interpreting effect sizes is still to apply Cohen's benchmarks from a half century ago. This has a range of negative consequences: underpowered studies, gross misinterpretation, dismissal of programs w/ "small" effects, & pursuit of unrealistic gains. /2

Social scientists have decried Cohen's benchmarks for decades, but his conventions have been remarkably difficult to dislodge. As @angeladuckw taught me, this is likely because people rely on simple heuristics when interpreting complex information w/ limited time. /3

I recommend interpreting effect sizes by "thinking fast AND Slow" (thxs @DanielKahneman) by striking a balance between practical considerations for interpreting findings quickly and important differences in the contextual features of individual studies. /4

Here are five broad guidelines for interpreting effect sizes across the social sciences. In the paper, I propose simple questions and corresponding interpretations for applying these guidelines. /5

Now for some controversy . . . as a quick approach, I propose a schema for effect sizes from causal studies w/ achievement outcomes. This includes new empirically-based benchmarks for effect sizes & costs. Reasonable people will disagree about what the exact cutpoins should be /6

Some big takeaways I came away w/ for interpreting effect sizes:

1. Many (Ed) interventions fail to produce effects

2. Larger effect sizes are not necessarily more important than smaller effects

3. Low-cost interventions are not uniformly more favorable than costlier ones /7

This paper reflects the collective wisdom of a large group of colleagues whose work & feedback shaped it profoundly. I am deeply grateful to @matt_barnum Howard Bloom @abrooksbowden @clconaway @Thomas_S_Dee @angeladuckw @AviFeller @CEDR_US @GuryanJonathan @douglasnharris . . /8

. . . Mike Goldstein @SusannaLoeb Dick Murnane @linzcpage @Todd_Rogers_ Nate Schwartz, John Tyler, @dylanwiliam & David Yeager for their support. I'm also indebted to my amazing grad RA fellows @AlvinCtweets & @abolves who helped me in every phase of this project. /9

As social scientists, we have to do a much better job interpreting & communicating the effects we find if we hope to influence policy. We also need to help the next generation of scholars develop these important skills. I hope this paper contributes to these efforts. [END]

P.S. . . . Looking for a short, accessible overview of my paper on how to interpret effect sizes? @CJRauchEduc has you covered with a great article in @SchoolsWeek.

schoolsweek.co.uk/how-to-interpr…

It's common to translate teacher/charter effects as "closing the Black-White achievement gap after X years" (i.e. B-W gap/ES). This requires strong assumptions about the independent & cumulative nature of ES that seem implausible especially given fadeout.

#econtwitter thoughts?

@NeeravKingsland has distilled my paper on effect sizes into a clear and accessible blog post. He applies the framework to interpret and translate @douglasnharris & @ERA_NOLA's findings on the effects of the New Orleans education reforms.

https://twitter.com/NeeravKingsland/status/1134075533426384896