Thread by @Nate_Cohn

I don't really agree with @NateSilver538's glass-half full perspective on the 2020 polling miss
fivethirtyeight.com/features/the-d…

I'd boil down the disagreement to one fundamental thing: I think low systemic bias is far, far more important than important for thinking about the polls than average error, while I think @NateSilver538 looks a lot more at average error

You can see both the magnitude of systemic bias in this chart, along with the case that there's a trend toward greater systemic bias. And fwiw, I think the D+5 bias is probably mitigated by some 'nonpartisan' firms that, tbh, aren't so nonpartisan or above the board

Now, should you care more about average error or systemic bias?
I think you should care way more about systemic bias.
Poll averages can be fine with high avg error, so long as bias is low.
But even the lowest avg error on record--4.8 points--would be horrible iff it was all bias

There are fundamental, underlying reasons why the risk of bias is increasing, while error stays low.
The quality of the survey data is decreasing, as response rates decline and polls go online.
Pollsters compensate by weighting, forcing industry-wide bias-variance tradeoffs

What's a bias-variance trade off?
Well, often times you can make a choice between reducing error at the expense of inducing possible bias.

Take party ID. Let's say you get back that's D+10 and Clinton+6. You can weight it to D+5 and Clinton+2 probably reduce error.
But if I always weight to D+5/Clinton+2, and the electorate is D+2/Trump+1, then *on average* my polls are biased by Dems to 3 points

Virtually every non-phone pollster is making choices like this. It does keep their error down and it can also reduce bias. But it also basically makes it inevitable that every election includes some systemic bias, especially since these choices are similar

If we get to the point where the typical systemic error is greater than 3 pts or so, I think that's a big challenge. The whole range of presidential election results in my lifetime is, what, R+2 to D+7? 3 pts, either way, includes most natl outcomes

As a result, I don't think the fairly typical average error in 2020 is really worth looking at tbh. I'm not saying it's irrelevant, but it kind of is!
If there's no systemic error, the polling averages will be great and the big picture story will be sound, even if error is high

The FiveThirtyEight pollster ratings, on the other hand, are almost entirely constructed around pollster average error.
The ratings basically don't care whether you're biased; the assumption is that they can get rid of the bias with House Effects

That's great for the world of 2006, when you could assume the industry-wide sample/choices are more-or-less nonbiased.
Today, you'd rather have a rating based on whether a firm is prone to systemic bias--though it's basically impossible to construct with the available data

Whether a pollster is prone to systemic bias really boils down to two things:
--does the poll make heavy bias-variance trade offs? lvie very rarely; ivr/online all the time
--is the underlying sample unbiased? this was live's case a few years ago; now it's at least in question

FiveThirtyEight is more or less acknowledging that this is happening, indirectly. Whatever qualitative advantage live phone samples still enjoy is no longer making up for their lighter weighting schemes, at least in an electoral context

But this has some long term consequences for the FiveThirtyEight scheme as a whole, since a world where live phones can be systematically biased is one where FiveThirtyEight can't reduce bias by calibrating its House Effects to a particular kind of survey

One last way to think about it and then I'll shut up.
Polling averages can work for one of two reasons: a) poll samples, industry wide, are more-or-less unbiased after basic weighting; b) pollsters, industry wide, can eliminate bias in ways that cancel out in the aggregate

I think B is a very scary world for poll aggregators. A lot of the underlying samples can be biased the same way. A lot of resulting choices will be similar (try past vote!). And a lot of the choices are calibrated to expectations (aka herding, which risks systemic bias).