Thread by @rp_griffin

#AAPOR ICYMI

@johnmsides and I examined major explanations for polling error in 2020.

The big takeaway:
Evidence that highly engaged Democrats were a big contributor to the problem in 2020. Reweighting by in-cylce measures of participation substantially reduced error.

Taking it from the top:
pre-election polling in 2020 was...not great.

One line of thought people had directly after the election:
-non-response by those low in social capital/trust might be causing these errors
-reweighting by some measure of social capital/trust might reduce error

Replicating some of the work Pew had done comparing prob/non-probability samples, we examined how this weighting scheme would affect the overall accuracy of a sample.

Short answer: Potentially helped reduce bias, but the effect was rather small.

The effect is even smaller on political variables. Has a negligible impact, if any.

Didn't explore this follow-up point by @davidshor. May be something we look at in the next iteration.

An alt explanation for '20 polling errors: Too many highly engaged, loyal democrats.

As proxied by participation in the '20 presidential primary, we find robust evidence for this. Too many primary voters and dem primary voters (even after weighting by '16 vote).

Reweighting by primary participation does a lot to reduce error. That's true using both voter validated participation in the primary or self-report.

Probably not a fluke either - there's a pretty consistent effect over 8/10 pre-election waves of the Nationscape survey.

This leads us to some tentative conclusions...

BUT also some substantial questions that are still up in the air.

Uncertain how generalizable these results are (though I have quietly heard other folks had these same primary participation biases) and uncertain how actionable this is in future elections.