Thread by @balexanderstats

I’m excited to be at #aapor this year. The first session I’m attending (and live tweeting) is “Assessing the Polls: Measuring Bias and Vote Choice”.

The first presentation is “It's Not Personal: Evaluating the Impact of Asking for Voters By Name” presented by Travis Brodbeck and Madeline Harland of @SienaResearch.

Sienna college polls have been some of the most accurate and are based on the L2 voter file and a likely voter model to determine the likelihood someone will vote. #AAPOR

Travis points out the challenges of phone polls which is the primary way Siena college does their election work. It can be difficult to get respondents to pick up. #AAPOR

One metric Travis mentions is important is the ratio to respondents reached to a survey complete. As an experiment compared they asked for voters by name or did not ask by name. #AAPOR

Next are plots analyzing the effect of asking by name on demographics in some states. The change in composition by gender and race, and party composition. There are differences across states. Ohio had minimal change, but other states like FL, NY, TX, WI had more change. #AAPOR

Now Travis is reviewing what the next steps are. Travis proposes the use of quotas to control the demographic composition while optimizing costs and response rates. #AAPOR

Next we have “Comparing Nonresponse Patterns in the 2016 and 2020 American National Election Studies” by Maxwell Allamong (presenter) and Sunshine Hillygus from Duke University. #AAPOR

First Maxwell shows the results of the 2020 ANES which had the largest Democratic bias in decades which was similar to other polls in 2020. #AAPOR

Now Maxwell is discussing nonresponse follow up studies conducted in the ANES to better understand if there was a difference between non-respondents and respondents and possibly identified correction factors. #AAPOR

Now the details of the nonresponse study is revealed. The 2020 nonresponse study had a higher response rate and larger sample size.

#AAPOR

Some key takeaways are nonrespondents are less likely to have a college degree or internet at home and some non-voters are missed. There is also more concerns about privacy and less social trust in nonrespondents. And nonrespondents were less likely to vote for democrats. #AAPOR

Next is are the results of models to predict vote choice based on different demographics and attitudinal preferences. Which has similar results to the univariate analysis. #AAPOR

Next there is an analyze of response rates among Republicans by vote choice. Republican who voted for Biden where more likely to respond than Republicans that voted for Trump. #AAPOR

Next is a plot comparing differences in disliking paper and web surveys. There is evidence that paper surveys might better reach Trump voting Republicans. #AAPOR

Now Maxwell is providing conclusions. He mentions that there is evidence of nonresponse bias and that Biden voting Republicans are likely being over-represented but changes in mode but help to better target individuals. #AAPOR

Next we have Dana Popsky from @pewresearch presenting “Did I Say Something? Perceptions of Political Bias and Attrition on Survey Panels”. #AAPOR

Dana first provides an overview of the American trends panel which is a probability panel where many of the polls Pew runs is conducted. #AAPOR

Next Dana reviews some reasons why people leave the panel such as lost interest. Dana mentions that at end at the of surveys respondents are asked if the questions are politically neutral. #AAPOR

Next Dana is showing the results across various surveys if survey was politically neutral which is that most people think they are neutral. Republicans are more likely to think the survey is one sided but depends on the issue. #AAPOR

For example a survey that included questions about January 6th including classifying it as a riot were more likely to be perceived as not politically neutral. Gender identify or environment surveys also have more people viewing them as not politically neutral. #AAPOR

Now the average percent of surveys a respondent that was politically one sided was about 6% among all panelist. But different demographics have different perceptions. #AAPOR

Groups with significant differences include Black panelists, panelists 18-29 years old, Republicans, and those with less than a high school education. #AAPOR

Next Dana is looking at the relationship between perceptions of political neutrality and attrition measured by the annual profile survey all panelist surveys. 92% of panelists were retained but there were some variation by perceptions of politically neutral. #AAPOR

Next we have my @jcbjackson from @ipsosus (where I also work) His first presentation is “Uncovering the Gaps: Examination of Differential Non-Response in the Ipsos-Fivethirtyeight 2022 Midterm Panel Study”. #AAPOR

This presentation is joint work with @FiveThirtyEight Chris explains that the purpose of this presentation to look at drivers of the past bias in election polls. This study uses Ipsos’s KnowledgePanel which is an online probability panel. #AAPOR

This study has a sample size of about 2000 in the first wave and about 1500 in the follow up waves. Next the unweighted demographic and cooperation rates are displayed which are pretty typical for online probabilities. #AAPOR

Over half of panelists took all 7 waves. Next Chris is explaining the analysis that is looking at differences in people who dropped out of the study. First are some slides showing attrition by demographics. There are differences in age. #AAPOR

There are some differences by the interaction of vote choice and party. Trump voters who were not Republican were more likely to drop out. Based on media consumption those who consume news from social media were more likely drop out. #AAPOR

Next are some analysis on differential nonresponse on the 2022 generic ballot, but the respondents were slightly likely to have voted for Biden and this slightly increased over time. #AAPOR

Next we have another paper presented again by @jcbjackson on “Get Off My Lawn: Evaluation of Alternative Population Parameters in Election Polls Using the Ipsos-New York Times Wisconsin ABS Mail Study”. This is joint work with @NYT.

This study is an experiment of a mail survey in Wisconsin compared with a KnowledgePanel survey and a @SienaResearch phone poll to see which methods perform the best. #AAPOR

The mail survey had 23% response rate and 1610 responses. There was a KnowledgePanel survey with about 1200 panelists. Next the demographics of the mail survey are compared to the benchmark. #AAPOR

The ABS survey had some differential nonresponse typical of surveys but weighting by demographics plus a likely voter model did fix some of the bias. Voters were overstated which is common. #AAPOR

Next some non-traditional benchmarks were used ranging from having a passport or owning things like a gun, ATV, boat. #AAPOR

Both the mail survey and KnowledgePanel overstate voter participation. Next a comparison of ABS and KnowledgePanel on the non-traditional benchmarks. KnowledgePanel does slightly underrepresented more individualistic behaviors. #AAPOR

Next some experimental weights based on the ABS sample to try to improve representativeness in KnowledgePanel. The best performing weight is a 2020 vote weight but the non-traditional benchmarks have a limited impact. #AAPOR

Chris points out that the biggest is is representing political behaviors well. Surveys tends to overrepresent politically engaged voters. #AAPOR