Thread by @NateSilver538

Our new pollster ratings are up!

They've been updated to reflect the results of the 2020 general election + the GA runoffs.

Also a shiny new interactive. Fivey Fox makes a cameo appearance.

Here's the link. I'll discuss some key findings in this 🧵.

projects.fivethirtyeight.com/pollster-ratin…

Note that we now have pages for individual pollsters. So you can see exactly which polls made it into the rating for each polling firm. Basically this means every poll within 3 weeks of an election since 1998!

projects.fivethirtyeight.com/pollster-ratin…

Or if you want to go even deeper, you can find the entire database on GitHub. We strongly encourage people to use this database for academic research, etc. A LOT of hours of gone into building and maintaining it.

github.com/fivethirtyeigh…

The next key link is to my feature article, "The Death of Polling is Greatly Exaggerated", which argues that the death of polling is greatly exaggerated.

fivethirtyeight.com/features/the-d…

Based on average error, the 2019-20 cycle featured decidedly mediocre polling: an average error of 6.3 points. That's the 3rd worst out of 12 cycles we've tracked. At the same time, as you can see, it's not an outlier exactly, either. It was well within the range of "normal".

The polls actually did pretty well in "calling" races correctly, as 79% of polls identified the right winner, which is the same as the historical average. Polls got 48 of 50 presidential states right, correctly identified that Democrats would win Congress, etc.

Is that the best way to evaluate polls? No. It's better to look at the margins. The D wins came by far closer margins than polls projected. Still, the media tends to judge polls by winners and losers so the 💩 the polls got was a tad inconsistent with how they're often evaluated.

But here's the problem that's much harder to excuse. The polls had a big Democratic bias (bias meant in a statistical sense). Republicans beat their polls by 4.8 points on average! The bias was actually larger in Congressional & gubernatorial races than for Trump (4.2 points).

That's the biggest bias in either direction in the cycles our pollster ratings cover (since 1998). It's likely that some earlier years, certainly 1980 and probably 1994, would have had a bigger bias if you extended back that far. So not unprecedented. But still ... not ... good.

don't think we should necessarily expect that polls will continue to have a anti-Republican bias. Historically, the direction of bias is not very predicable as pollsters adjust, adapt, etc. But I do think we may continue to see systematic polling errors in BOTH directions.

In an environment where politics are highly nationalized and polarized, you don't really have "50 separate contests" for the presidency. For that matter, presidential and downballot outcomes are highly correlated. So if your polls are off in one race, they may be off everywhere.

The other big finding is that we no longer see a clear rationale to give live-caller polls a higher grade by default in our pollster ratings. For one thing, they haven't particularly outperformed other methodologies.

For another thing, it no longer really makes sense to classify entire *polling firms* by their methodology. Lots of polling firms mix-and-match methodologies, change them in midstream, etc. Methodology is a characteristic of a poll, not the pollster.

That doesn't mean that quality doesn't matter. We find that pollsters that participate in professional transparency/data-sharing initiatives continue to get considerably better results. The pollster ratings will continue to reflect this.

Something else worth mentioning—it's common sense but it shows up in the data—is that you should be mildly distrustful of pollsters without a track record. As a rule of thumb, it takes about 20 polls before you can have much confidence that what pollster does is working.

Finally, here's how the most prolific pollsters fared in the general election.

The best-performing pollster was AtlasIntel!

Second was Trafalgar!

Yeah, they incorrectly had Trump winning a few states, but they were close on the margins, and that's the better metric.

And ... one last piece (for now) of pollster-ratings-related content. A podcast! If the view I expressed in the article this morning struck you as a little too optimistic about the future of polling, this is shaded a bit more pessimistically FWIW.

https://twitter.com/galendruke/status/1375220916838350848?s=20