There is a new CNN poll out today with approval at 35, disapproval at 58. That may seem a surprise with polling averages at 43 (HuffPo) and 42 (RCP) or 39.4 (538) or 44 (me, sort of).
It’s not. And it raises an issue.
Let me elaborate. 1/x
It’s not. And it raises an issue.
Let me elaborate. 1/x
We have always known that house effects, due to methodology, sampled population, question wording, weighting etc etc are part of surveys, and these can be taken into account. Likewise with mode effects— the method by which data are gathered. 2/
In polling on Trump, live phone polls (which include cell phones) have consistently given lower approval ratings than have internet based surveys which in turn have been lower than IVR (robo) which now usually include an internet component to compensate for lack of cells. 3/
As a purely practical matter so long as the mix of polling modes is pretty good (some of each mode each week) and no mode dominates recent polls these effects make relatively little practical difference. Dropping one particular poll matters little for the trend or average. 4/
If one wants to make the president look better one can pick IVR or to a lesser degree Internet modes. If one wants a lower approval, choose live. But if one wishes to represent the wide range of results across pollsters without prejudice to mode, then use them all & average 5/
Recently there has been one important shift in the balance among modes. The end of Gallup’s daily data in January and its replacement w weekly results has reduced the number of live phone polls in a typical week. Rasmussen, an IVR poll, now is the only daily reading,
When we had both Gallup & Rasmussen each day (some thin this to every 3 days) we had an even balance of daily live vs IVR polls, so while one was higher and one lower, the average between the two wasn’t that far from the overall trend line. 7/
So how much do the trend lines differ by mode? This much. 8/
That’s quite a gap right now. All modes agree there was a rise in Trump approval in December and January. That is important as it shows the modes differ in level but not so much in trend. This validates different modes to a point, but accepts mode effects as real. /9
The mix of survey organizations within mode is quite imbalanced. Among IVR Polls here, Rasmussen contributes 276, PPP 13 and Gravis 1. So “IVR” mode is essentially the house effect for Rasmussen with little additional influence from other pollsters. 10/
Among the Internet polls however, there is very good balance among YouGov (58), MorningConsult (56), SurveyMonkey (57) and Reuters (56) with a few from others. 11/
This is good because no single pollster dominates the mode and we can also see how much variation within mode there is. 12/
Here is a look at trends by individual pollsters. Note the difference by mode, and the common movement up or down between successive polls. (See esp recent live polls) 13/
In live polls with recent polls, ABC, CNN, Fox, Marist & Pew each show a rise followed by a decline in their latest poll. Q-Poll is flat. Of the Int Reuters & MorningConsult dip while SM & YouGov are a bit flatter. (Gallup is daily only in the figure. Their last weekly is 37.14/
So what is the upshot to all this? First that the mix of live polls has changed with the loss of Gallup Daily, and that needs to be recognized in simple averages. I expect 538’s average is a bit lower because they remove house effects while RCP and I (have) not.
15/
15/
The IVR category is dominated by Rasmussen, so we are not seeing a mode effect so much as a house effect alone. All the more reason to incorporate house effect estimates and adjust accordingly. 16/
The relatively decline in frequency of high quality, live phone, polls and the fact that we don’t get them every weeks, as we do with Internet or IVR, means that we can have one week dominated by Internet or IVR and another with good live poll representation. Care is needed. 17/
The bottom line is when you see a trend for all polls, check if house effects have been incorporated in the estimates. It is also good to break out results by mode, so see the range of results. /fin
