THREAD.
1/ I was looking through twitter today and saw how much people mistrust forecasts of the election. We are pretty bad at thinking probabilistically and it seems as if once a candidate is somewhere over 60% we default to assuming they are 100% to win
1/ I was looking through twitter today and saw how much people mistrust forecasts of the election. We are pretty bad at thinking probabilistically and it seems as if once a candidate is somewhere over 60% we default to assuming they are 100% to win
2/ so if they lose we declare the forecast (or the polls) wrong! This is, of course, aided and abetted by a lot of pundits. It’s very frustrating because I am not sure how we can fix the problem.
3/ Certainty is what people want. They don’t want to hear that one candidate is 25% and the other 75%. They want to know for sure. And the pundits are often more than happy to give them what they want, offering more certainty than the data offers.
4/ Here’s a good example of that desire for certainty:
5/ Pundits really are a lot of the problem here as they express things with way too much certainty and get way to excited by a single poll showing a race tightening. They are often looking for attention grabbing headlines rather than the truth so they cherry pick polls that ...
6/ tell a good story and then they over-hype those polls, which are often outliers. If a forecast shows a heavy favorite, the pundits act as if it is 100% that the favored candidate will win, reinforcing that error in the public.
7/ That is why this tweet is so funny:
8/ You can see that error in a lot of the chatter from people saying that we should ignore the forecasts because @fivethirtyeight had Trump slightly less than 30% to win the presidency on 11/8/2016 and <GASP>
9/ he won so obviously the forecast was wrong because less than 30% obviously means ZERO PERCENT so now we should totally ignore 538’s forecasting around the midterms.
Here’s one example of that but a quick search pulls up a ton of them:
Here’s one example of that but a quick search pulls up a ton of them:
10/ I saw some of this reinforced in this @PeterHamby piece in @VanityFair which dunked on a lot of the punditry but also basically says we should not trust the polls. Polls are not meant to be predictive! They are inputs into a forecast or a prediction market.
11/ The author cherry picks a bunch of past races where the polls were way off the final result. First, I can also cherry pick a bunch of races where the polls were dead on (like the prediction that Clinton would win the national vote by 2% to 3%).
vanityfair.com/news/2018/11/s…
vanityfair.com/news/2018/11/s…
12/ Cherry picking when polls miss by a lot to claim they are bad data is kind of like saying the line in an NFL game was 7 but one team one by 12 so lines are stupid! The point is that POLLS ARE DATA NOT FORECASTS.
13/ They are like a mock vote taken the day of the poll and how much signal the poll provides for predicting the final outcome depends on the polling methods. That is why @fivethirtyeight grades the polls.
14/ Also polling quality is generally better quality at the national level than the state level (even while state polls are better inputs into forecasts of a national election in the US) and, even at the state level, polling quality varies significantly
15/—partly due to differences in the size of a state and/or the homogeneity of the state. Here’s a great piece explaining this from 2016.
fivethirtyeight.com/features/is-a-…
fivethirtyeight.com/features/is-a-…
16/ We want someone to tell us who is going to win but no reasonable forecast can really do that. In our heads, once there is a heavy favorite, we default to processing that as a prediction of who will win for sure. Pundits oblige by offering certainty where none exists.
17/ And then when a favorite is upset, the public declares the polls wrong and our trust in expertise corrodes, not because the forecasts were wrong but, rather, because the public and the pundits interpreted what the polls/forecasts meant poorly.
18/ I think we would all be better off if we all participated in prediction markets like @PredictIt which would force more probabilistic thinking and a little more Thinking In Bets, otherwise known as “put your money where your mouth is.”
19/ To put this into context, if I offered you a bet on a coin that would land heads 28% of the time, would you bet your life on the coin landing tails? NO!
20/20 Once I put it in the context of a bet, the uncertainty in the outcome bubbles to the surface and we can see more clearly that it isn’t a sure thing. END OF THREAD. #2018Elections #Polls #Forecasts
