Pity the poor pollsters
Yes, the title is meant to be sarcastic.
But only somewhat. Whatever you think of pollsters’ motives—are they trying to get it right, or are they skewing results to influence public opinion?—they do have quite the challenge in predicting voter turnout by party.
If they’re presenting rigged polls, how much Democratic turnout is just right? How much is too much? And at the end of the day, don’t they care about their own reputations for accuracy, and how they will suffer if they present biased polls that fail to predict the result? Or is it okay because they’re trying to create a certain result (hardly a foolproof proposition, that)?
But for those pollsters who are trying to get it right, what model should they use for their turnout predictions? 2008 or 2010? That is the question. And furthermore, even if pollsters ask respondents how likely they are to vote, and then see what relative percentages of Democrats and Republicans say they are likely and then weight the poll results accordingly, does this represent reality when they’re getting only a 10% response rate? Isn’t it quite possible that the responders already represent a skewed sample that’s likely to differ from the population as a whole on the question of motivation to vote?
Here are some sobering figures on response rates:
Note that even compared to a few years ago it’s significantly worse. What does this mean? It’s not just the result of the inclusion of cell phone users, either:
The most recent decline results partly from the inclusion of cellphone numbers in its samples in order to reach the rapidly growing number of American adults who have a mobile phone but lack landline telephone service. But the Pew Research landline response rates have also fallen (from 25 percent in 2007 to 10 percent this year) and are now only slightly higher than the response rates currently achieved with cellphones (7 percent).
For what it’s worth, my guess would be that cell phone users would tend to be younger and therefore more likely to be Obama voters, and so their slight under-representation could mean a slight underestimate of the Obama vote (or the Ron Paul vote). But far more important is the more general question of whether the responders represent a random group, or whether they differ from non-responders in significant ways, and if so what those ways might be.
I challenge anyone to come up with an answer, because you can’t ask the non-responders. They are, by definition, not talking.
The technical term for the problem is non-response bias. Here’s a discussion of the phenomenon as it relates to surveys of all kinds, and the following describes some ways to try to control for it:
Suppose again that additional demographic or database variables are available for all members of the targeted sample group. These variables are used to create sub-groups containing respondents and non-respondents. Weights are then calculated based on the proportions in each sub-group and applied to the respondents to reflect the total sample population. Comparisons on key variables are then observed between the unadjusted and weighting-class adjusted respondents. If clear differences are detected, then non-response bias is assumed to be at fault and the weighting-class adjustments are used as they provide results with less bias. Poststratification is another technique similar to weighting-class adjustment, except that the procedure uses population counts instead of the total sample counts. The downside to these techniques is that they assume that the differences between respondents and non-respondents are captured in the subgroups, and that there is no rule of thumb for comparing adjustments to determine which to use.
As I said, pity the poor pollsters.
Once again let me say that I’m not assuming that all, or even most, pollsters are trying for that sort of accuracy. My point is that the ones that are attempting to get it right face a formidable task. Of course, they knew that when they went into the field. But back then, the response rates were so much better that non-response bias wasn’t considered such a big deal. Now that it’s grown, there’s a much bigger chance that it represents a very big deal.
Many people in the blogosphere focus on the dilemma of party proportion as vitally important. And there’s no question that it is. But that’s an easier problem to see; after all, one can view (for the most part) the party affiliation each pollster uses, even if it’s nearly impossible to tell if those proportions would be correct for this voting year (I might also mention that there’s the problem of people calling themselves “independents” but who really are disaffected former members of a party who vote pretty reliably with that party nonetheless). Non-response rate bias is much more hidden, unknown, and probably more difficult to control.
I don’t know if I’m especially typical of anything, so the following represents a sample of one. But I’m an older person who has only a cellphone, my motivation to vote is at the highest level (would crawl over broken glass etc. etc.), and I probably would not answer a pollster.
Not that I’ve ever been asked.
[ADDENDUM: Ed Morrissey weighs in on polls and party affiliation.]
I am like you, neo, but I don’t even answer my phone unless it’s someone I know! And it rings constantly, all day long!
1. I’ve never in 59 years been polled, not by phone or in person nor by Mail. This is why I don’t believe in polls and their results, it’s not real each poll serves it’s own agenda. Therefore it’s suspect.
2. Many people fear to answer when a future administration with more power and less regard for free speech may just seek out the polled for their views which do not fit the agenda of the winners of the election or the culture war for revenge or elimination.
3. I fear my answer may just affect me in years ahead by my belief that free speech will protect me from the ones who wish my views be declared “hate speech” and me being punished.
POTUS Obama wishes my and every ones free speech curtailed to suit the religious views of a violent and intolerant religion per his speech at the UN recently.
CathyB brings up a good point. Caller ID may be an even greater obstacle to pollsters than cellphone-only users.
OldereandWheezier: I think that caller ID has to be a large factor. But that in and of itself wouldn’t matter, if those who screen calls that way are random. The important question is whether those who screen their calls that way differ in some other important way from the population as a whole. I don’t think anyone knows.
Neo:
I don’t remember ever having been polled until this year, and I have now been polled at least a half dozen times. I live in NV, a swing state (that probably explains part of the interest in my political persuasion), but I am also being exposed to a type of poll I had heretofore not experienced.
What is new to me is the push poll, a large number of which have come my way. You know the kind: “Do you plan to vote for Dean Heller even though he has voted twice to end Medicare, or for Shelley Berkley, who is committed to increasing Medicare funding?” Some are more subtle than this, others are insulting to the intelligence.
The problem for me is that I have little confidence I recognize all of them for what they are, and even less confidence that my neighbors will. The obvious ones just get hung up on; the more subtle ones are treated as an honest effort. This is probably a mistake: I should either treat all telephone polls as one or the other and furthermore, either answer all polls honestly or answer none.
The point is, the polling industry is going the way of the American educational establishment. An honest attempt to accomplish something of value has been recognized as a method of imposing someone’s will on the rest of us, and so has lost its value.
Which makes me gnash my teeth in impotent rage. Some days I wake up thinking the only answer for the mess we’re in is to hope the Mayan calendar has it right.
Sweet Meteor of Death 2012: The Solution to Politics As Usual. Forever.
Guess I’m the odd man out because I want to answer the pollsters. To get some points for the Right side.
I had never been called before either until Virginia became a swing state.
Neo: who would be using a 2010 model for polling? Mid- term election years skew older/ more Republican. I’ve read that the model will be more like 2004, a base turn-out election.
I still have a landline and ignore all calls that come from suspected telemarketers (that is, any “unknown” name or number I don’t recognize).
However, during the 2004 election I actually received a call from Quinnipiac (sp?) Said so on the caller ID. I knew it was a pollster and was happy to pick up and participate. I took much glee in telling them that I was a registered Dem, planning to vote GOP.
Since then I have made the official party switch (but only because I can’t vote in the primaries as an independent in PA).
KL Smith: who would be using a 2010 model? Someone who thinks that enthusiasm for Obama is tepid, and objection to him strong.
It’s anybody’s guess, but that seems as good a theory as any.
And looking to polls to answer the question of how enthusiastic voters are isn’t all that helpful if they’re using the wrong model in those very same polls.
This poll is a bit outdated (it’s from July), but it shows a diminished amount of enthusiasm among Democrats. And this CBS/NYT poll from around the same time shows GOP enthusiasm rising. I can’t find any newer polls on this in a very quick search, although I imagine there might be some.
I’m in PA, and a couple of months ago I was getting called for polls a lot. Then I donated to Romney, and the calls for more money started coming almost daily. Added to this is that charities have unwisely chosen this time to start calling also. Breast cancer called three times last week – twice in one day! Now, I thank God for caller id, and don’t answer the phone unless it’s a friend or family member.
Part of the reason for declining response rates is over-polling. With a dozen calls a day from political parties, pollsters, and telemarketers, some peoples’ willingness to participate declines. Sort of like the Heisenberg uncertainty principle – the more you try to measure, the less reliable your results.
Wellll, an oversimplification, but here goes.
Polling can be broken down into two parts:
-1- Do the sampling, and
-2- get the appropriate weighting on the subgroups within the sample.
Item -1- consists of categorizing those polled; are you male or female, what’s your age, what’s your 2011 income, what’s your race, etc.
Item -2- consists of weighting the categories, and this is where using 2010 or 2008 (or some other) turnouts come into play.
I’d say pollsters will easily get item -1- right, as it’s little more than bookkeeping and STAT 101. I’d say further that they very probably ^do^ get item -1- right — but it’s only the first step in the (oversimplified) two-step process.
The issue is in item -2-, concerning which little more need be said that hasn’t already been said in this forum.
There’s no way of knowing whether a late September poll for a November 6th election is correct; there’s nothing with which to compare, except for other polls. So we can say, as an example, Pew has Obama up by 8 percent, whereas the others have him up by 4 percent. Maybe Pew’s correct (in this example), and the others are wrong — but more fundamentally, who’s correct and who’s not?
—
I saw something last night, that referred me back to a RealClearPolitics web page from 2008. Pew had Obama beating McCain by 14 percent two weeks out, and by 15 percent one week out. At election time, Pew had it about right, which is to say, what? — was it 6 or 7 percent?, in the only tally that counts and can be used to compare.
Does anybody this side of sanity believe McCain closed that gap to that extent [or was there never such a yawning gap to begin with?] in the week before the election?? — especially when no other polls were registering that huge a difference?
My point: what were the Pew folks smoking (and then suddenly sobered up)? *OR*, was there an agenda lurking behind item -2-?
Okay, time to quit . . .
MJR: in those Pew polls from 2008, did the percentage of Democrats sampled change?
Neo asked (3:07), “MJR: in those Pew polls from 2008, did the percentage of Democrats sampled change?”
Neo,
The party breakout of those sampled is not included in the published data (although, obviously, it’s out there ^somewhere^ if one digs deep enough). For what it’s worth, here’s the web page I referenced:
http://www.realclearpolitics.com/epolls/2008/president/us/general_election_mccain_vs_obama-225.html
Many poll results from many numbers of weeks out.
Neo: agree enthusiasm for Obama is down, etc. but, mid- term turn out is always lower than presidential year elections.
Sean Trende at Real Clear Politics, Jay Cost at Weekly Standard, and Rasmussen Reports have the best unbiased info on polling.
And of course the one that matters the most is on Nov 6.
I think Presidential polling as currently implemented suffers from a serious case of over thinking and complexity.
It makes about as much sense as figuring out how many Americans eat broccoli, by first weighing broccoli’s sales 4 years ago and deducing a portion of its current support must be negated or added to to arrive at an accurate figure. Then of course you have to know the race, gender and socio economic status of the broccoli eater to determine if he actually buys the broccoli he eats.
WTF?
How about just ask 1000 random people in polls on a repeatable basis, who they support for President or whether they eat broccoli and trust that the sheer mathematics of it makes all the other BS obsolete?
Where am i wrong here?
I’ve been polled a couple of times in past years, but recently I became so sick and tired of telemarketing calls that I started leaving my answering machine on all the time. (I still have a landline and no cell phone.)
I get 4-5 calls per day, and most of them are hangups. I don’t know if they are pollsters or not. I would decline to participate in polls now. I believe that polls are more about molding public opinion than reflecting it.
The things you learn on the internet. I’ve read more poll analysis in the last three weeks than I ever wanted or needed to. You’re not the only one who is analyzing the polls, neo.
It is plain that polls can be a good way to gather information, but they can also be massaged in many different ways to give desired signals to voters. Thus, most of us don’t trust the polls, but like all humans, we want to be able to see into the future. The polls give the appearance of doing that. Is there anything about the way the game of politics is played in the U.S. that hasn’t been corrupted?
Fifty-two years ago I didn’t really worry about whether Nixon or Kennedy won. I expected the country to continue pretty much as it had. Today is much different. Is it because I’m old and senile? Or have things really changed in those years. I think the latter.
Kennedy ran as a conservative Democrat. In some (if not many) ways he managed to run to the right of Nixon. Nixon’s reputation was as an anticommunist, but Nixon himself was more of a go-along-to-get-along statist domestically and internationalist in foreign policy.
I agree, the country was going to continue along the way it had been, regardless of who won in 1960.
I think 2012 is mi-i-i-ighty different. (You ain’t senile.)
Neo wrote: “there’s the problem of people calling themselves “independents” but who really are disaffected former members of a party”
and likewise people who claim to be undecided who really have already decided. Dick Morris’ clarifying question is that being an “undecided” voter is like being asked to answer the question “Will you be married to the same spouse next year?” Any answer other than “Yes” is an equivocating statement about the voter’s relationship with any incumbent, and presumably why most “undecideds” historically break for the challenger.
MJR @ 3:01 pm wrote:
Remember pollsters earn there street cred by getting the correct results at election time, but they earn their income in the time before the election when they are constantly adjusting changing variables.
SteveH @ 5:14:
Because “random” is probably not truly random. All kinds of biases enter into whom to call. If you choose any one person from each of the 50 states, that is no longer random; you’re choosing one person per state. If you choose one person from every other page of the phone book, THAT isn’t truly random. In fact if you can insert the phrase “if I choose . . .” as the lead-in to any criterion, your choosing has negated a truly random effect.
“T”, 12:23 am: Very, very well put; succinct!
—– —– —– —– —–
Neo (and distinguished readers): Allow me to call your attention to this piece:
http://www.nationalreview.com/corner/328544/romneys-new-strategy-stanley-kurtz
Romney’s New Strategy
By Stanley Kurtz
NATIONAL REVIEW ONLINE
September 25, 2012 9:59 P.M.
Like a lot of other things in our world some of the assumption on which polls are based almost certainly have changed. Like are polls even trying to be honest or are they postmodern? It was said at the time that the polls blew ’48 because they used the phone book and undersampled folks who didn’t have a phone – who it was assumed were more likely to vote for Truman. (Hey. maybe that was Dem spin 40s style?) Now every generation uses the phone differently. Stats and social science methods have gotten better, but I’m postmodern too and would have no trouble practicing alternate narratives with pollsters. Still people like Jay Cost, who has feet on the ground, seem to take polls seriously. Still, I have never doubted them before – particularly in aggregate like the RCP average – but this year I feel like the ground is changing underneath our feet. I wont be surprised if they get it very wrong.
I believe that the most accurate polls are the internal, confidential polls done for the presidential candidates.
We can’t see the results of these polls, but we can see where the candidates are campaigning.
Obama is now campaigning in Ohio. Would he be wasting his time in Ohio if he was really seven points ahead of Romney?
“”Because “random” is probably not truly random. All kinds of biases enter into whom to call.””
T
I respectfully disagree. If you pick 1000 names at random out of a random American phonebook and repeat that 10 times, you’ll get an amazingly accurate result. Do it 100 times and you’ll get an undeniably accurate result. Do it 1000 times……
This isn’t rocket science. It’s just simple mathematics that doesn’t know what a bias is.
Lorenz Gude,
“. . . are polls even trying to be honest or are they postmodern?”
I’d like to think it’s both. IMO Rasmussen and Gallup (even under DOJ Lawsuit threat?) are interested (IMO) in an accurate aork product, at least to a point. PPP, on the other hand, has shown definite willingness to falsify (not just “massage”) its polling; remember PPPoversampled Republicans by +9 in the wake of the Todd Aikins controversy to convince him to stay in the race. A +9 Republican sample? In what universe?
The other point to make here is that just about all of the people who issue, analyze and study these various polls all exist within a bubble. The other night Brit Hume, whose opinion and insight I regard very highly admitted being dumbfounded as to just how Obama continued to “defy gravity” (his words) in the polls.
The point here is that even someone that insightful and who tries to be very objective is, IMO, unaware that he can’t see the forest for the trees. Implicit in his remark is the belief that the polls reflect the reality on the ground.
What if they don’t? What if the reality on the ground is really reflected by the results of unskewed pollsd.com and the Colorado study (i.e., Romney/Ryan leading and winning by ~53%)? IF this latter is true, then the answer to Hume’s gravity defying question is easy; Obama’s NOT defying gravity. He’s like a David Copperfield who makes it seem that he is levitating, but any onlooker knows it really can’t be done. This transforms Brit Humes’s question from the implied “How is he surviving all of the foreign and domesti meltdown so well?” to “how does he appear to float even though we know it is really not happening?”
Neo’s table in her essay would seem to support this latter interpretation (polls possibly becoming less predictive). It is in this inability to recognize such a possibility that even astute observers like Brit Hume fail beause they don’t recognize their own bubble.
. . . and this today from Jay Cost:
http://www.weeklystandard.com/blogs/morning-jay-are-polls-tilted-toward-obama_653067.html
Perhaps the most important quote from Jay Cost cited above (p.3):
If Jay Cost is correct, then this answers Brit Humes’ question about defying gravity. Obama is not. defying gravity, it is only a “David Copperfield effect” in which he temporarily (and under very limited conditions) appears to be in the lead.
IMO the flaw is in the use of any weighing based on party affiliation. People will cross lines in voting for President as witnessed by Reagan democrats and Obama republicans. Which makes weighing a rather useless step that pretends such things don’t occur.
The uncanny abscence of Obama bumper stickers this year tells anyone with common sense that a party affiliation model is of little or no use in this election.
SteveH,
You mention the “uncanny absence of Obama bumper stickers.”
IMO this is another “tell” that indicates something is going on that is not represented (at least not yet) in the polls.
I’ve already made th epoint that by itself it doesn’t mean very much, but when added to other like events that are occurring nationwide it implies that something is afoot.
I’ve cooperated with phone pollsters a few times over the years.
But the last one was about a week ago, to my cell phone, and I just flat out said no.
First, I was pretty busy with work-related issues.
Second, and more importantly, it was going to take 15 minutes plus to get through this one. On my cell phone.
I laughed at the lady when she told me how long it would take, after she followed up with “…was now a good time?”
I replied by asking if she knew that this was a cell number, and remarked that it was never going to be a good time. It was a cell call.
Oh, all very politely, of course. I’m generally not overtly mean with people who are just trying to do their job. (With managers, I’m not nearly as forgiving.)
But.
Y’know? – I pay for my cell minutes during the day.
What the hell is wrong with these morons.
davisbr: if you use US Cellular, all incoming calls are free minutes.
Just sayin.
SteveH: but actually (I don’t have time to find the link right now, but I’ve seen this statistic a couple of times) party affiliation matters a great deal this time. Something like 90% or more of Republicans plan to vote for Romney, and something close to that of Democrats say they plan to vote for Obama.
T: as far as that 9% Republican poll goes—well, Missouri does lean Republican. But not by 9%. According to this, more like 2%.
Neoneocon,
Agreed. As for my point several threads ago, there is unconscious bias, conscious bias and downright misrepresentation (aka fraud). I would place that particular PPP poll in the last category.
Low response rate is a huge issue because it depends on the willingness of the contact to complete the process. Not only is the number of questions important, but any perceived bias in the pollster can cause a non response. Given that most conservatives/tea party types consider pollsters (media sponsored) as part of the MSM, they will be reluctant to respond. My guess is this introduces a huge bias (10% or more?) towards the dems. Party affiliation may be a good marker for this.