Thread by @jakehofman

Excited to announce a new paper in PNAS today!

An illusion of predictability in scientific results: Even experts confuse inferential uncertainty and outcome variability



Joint w/ great coauthors @_szhang, @P_Heck, @MichelleNMeyer, @cfchabris & @dggoldstpnas.org/doi/10.1073/pn…

The TL;DR is that one of the most common ways we communicate scientific findings can lead people to drastically overestimate the size and importance of those findings. But there's an easy fix. Read on for more.

In many fields, there is a long-standing emphasis on inference (precisely estimating an unknown quantity, such as a population average) over prediction (forecasting individual outcomes). So it's common to see plots like this, showing means and standard errors/confidence intervals

Here, we show that this focus on inference over prediction can mislead readers into thinking that the results of scientific studies are more definitive than they actually are.

For instance, when doctors, data scientists, and faculty saw plots like these, they perceived the treatment to be much more effective than it actually was, often estimating that there was a >90% probability of superiority between conditions when the true effect was below 60%!

In contrast, when we communicated both inferential and predictive information side-by-side (adding individual data points alongside statistical estimates), participants had very well-calibrated perceptions of treatment effects. Here's an alternative version of the plot above:

And here's how big of a difference this made in terms of how the experts in our studies perceived these two different presentations of the same data, plotted of course with both statistical estimates *and* individual responses:

As @cfchabris likes to put it, putting dots on plots correct incorrect thoughts! :)

This "illusion of predictability" is likely due to readers confusing the concepts of inferential uncertainty and outcome variability and consequently mistaking precise statistical estimates for certain outcomes.

Our guess is that this specific issue, while centered around data visualization, reflects a broader concern around how science is done and how results are communicated, but thankfully there's a simple solution: communicate both inferential and predictive information side by side!