Thread by @jonmummolo

New @The_JOP by @seanjwestwood @SolomonMg @ylelkes shows probabilistic election forecasts like @FiveThirtyEight confuse voters & decrease turnout, mostly among Dems. It’s thorough and innovative experimental behavioral research.

A fan thread. (1/n)

Unlike polls that show candidates' expected vote share, prob. election forecasts convey the estimated probability that a candidate will win. Problem: folks don’t understand probabilities. This paper demonstrates severity of this confusion, and its political consequences. (2/n)

What I love about this paper is how it attacks the problem from so many angles. Not only does it feature a series of careful and novel experiments, it uses a ton of observational data to provide context for the study, clarifying the implications of the experimental results. (3/n)

For example, analysis begins by systematically describing context in which prob. forecasts are consumed using cable news discussion, Twitter shares, Google searches & more. Prob. forecasts are widely discussed, especially in outlets viewed by Democrats. (4/n)

Further, ANES data show Dems were more confident Hillary would win in 2016, and the most confident were less likely to vote. Perceptions of the race are important because voting is costly. If a landslide is expected, there is less utility in voting. (5/n)

A natural experiment shows influence of prob. forecasts: “transitory error” in 538's algo caused 2018 U.S. House predictions to spike. PredictIt & U.S. Bond market reacted. This alone could support a neat paper, but it’s presented here just to give context for main analysis.(6/n)

Do prob. forecasts distort perceptions? Experiment #1 asks participants to evaluate hypothetical election; supplies either a prob. forecast, a vote share (poll), or both. Crucially, both projections are crafted to convey the *same* underlying information in different form. (7/n)

Key result: “when faced with a high probability of winning, respondents reported vote share as if they expected a blowout. Yet in the condition that provided vote share, likelihood hovered around 50-50.” (8/n)

In other words, a high prob. of winning is perceived as a landslide election, even though that may correspond to a candidate leading in the polls by only a few percentage points. Prob. forecasts distort perceptions of the state of the race. (9/n)

All of this might be fine if these perceptions don’t affect voting. To test this, the authors conduct an elegant behavioral experiment with real money on the line (to avoid results driven by cheap talk). (10/n)

Participants engaged in hypothetical election. They got $ at start. Each randomly assigned to see prob. forecast & vote share conveying different states of the race. They got additional $ if their team won. They could vote to help, but had to pay small amount of $ to do so.(11/n)

Key result: “As the probability of winning diverged from 50-50, participants were less likely to vote...However, we detected no effect of vote share extremity on voting.” (12/n)

In other words, because people interpreted high probabilities as indicative of impending landslides, they abstained from voting. There were no such abstentions when vote share was supplied instead. (13/n)

Recall that prob. forecasts are mostly consumed by Democrats. Now think back to 2016. “To the extent that this experiment generalizes to real-world elections, the effects above are large enough to meaningfully alter turnout in marginal states.” (14/n)

In sum, prob. forecasts confuse people and lower turnout, more so for Dems than Reps. Effects are large enough to suggest meaningful impact on elections. And given that the same info can be conveyed as vote share, forecasters should just report vote share! (15/n)

Congrats to @seanjwestwood @SolomonMg @ylelkes on a fantastic study. It’s an instructive example of how to surround a really difficult research question and provide as careful and thorough an answer as possible. I learned a lot from reading it. (n/n)

journals.uchicago.edu/doi/pdfplus/10…

Ungated version here: dartmouth.edu/~seanjwestwood…