Puzzling polls
Two pundits have noted a puzzling discrepancy between the national polls and those in the battleground states.
First, Jay Cost:
On the one hand, Mitt Romney has built a narrow but durable lead in the national polls, averaging around a 1 percent advantage over the last three weeks. This has cheered the hearts of conservatives everywhere.
Yet, liberals retort, Obama has a lead in enough swing states to add up to 270 electoral votes, and that is really what matters.
What to make of this?
For starters, they cannot both be right. If Mitt Romney wins the popular vote by 1 percent or more on Election Day, the odds that he will lose the Electoral College are quite small.
Next, Sean Trende:
Put simply, the national surveys point to a Romney win, while the state polls collectively point to an Obama win. Both can’t be correct.
Both men then go on to number-crunch in order to underline why either the national polls or the state polls are almost certainly wrong. And they both conclude they haven’t a clue which it is. No wonder so many people on both sides are tearing out their hair!
Why are the polls so strange this year? I think it’s because we’re in the midst of a of polling transition. For years the response rate—the number of people who answer polls—has been shrinking, and it may have reached a point where it’s mattering more and more in creating poll variation of some sort. If only 9% of respondents cooperate, versus over 25% just a decade ago, we should become less and less sanguine about whether the respondents can be said to be a random sample.
It is often said that if a sample is large enough, randomness matters less and less. But that’s only true if the sample is truly jumungous, which does not occur with polls. Otherwise, even a very large sample is garbage in/garbage out if it isn’t representative of the group’s composition.
That’s why I think there’s so much weirdness going on with the polls this year. I would love to know if state polls generally have a different methodology than national ones, because such differences could go a long way to explaining the disparities between the two. For example, does one type of poll generally include cell phone users while the other type does not? Does one type of poll use recorded questions and the other type live questioners? Does one type of poll occur at a different time of day than the other type? I don’t even know what some other differences might be, but my guess is that they exist, and they could matter.
I know I get nervous when, in order to think my guy is winning, I have to invalidate the results of too many polls. “That one doesn’t count because the Democratic sample is too large.” That one doesn’t count because the poll is too small.” “That one doesn’t count because the pollster is biased.” And on and on and on. It’s been my experience that the losing side is the one that has to make more of these excuses prior to the election.
This year, though, both sides are making a lot of excuses—just about different things. That’s why my teeth are slowly grinding down.
Statistically the problem with the low participation rates is the danger of the poll becoming self selected. In the Pew study they concluded that all demographics were equally non participatory; however, I think the “Block Model” they used is more confirmation bias than useful. My own narrow sample space of family & friends indicates that conservatives are far more likely to screen calls. Anecdotal, I know, but in discussions I’ve seen in comments around the web it seems to have validity.
If you look at historical Polling data doesn’t it show a preference for Democrats? I’m going by memory here, but IIRC, the polls had Reagan down by significant margins in 80 & 84 just a week or two prior to election day.
Remember, it’s all about turnout. They’re demoralized, so ride right through them!
Here in SW Florida we have had early, in person voting since last Saturday. The polling station is open from 7 am to 7 pm
Yesterday, I went at 3 pm, figuring I would be okay; already Wednesday, and not the lunch hour. Nope. I couldn’t find a parking spot and it was going to be a two hour wait! (Line all the way around the outside of the building – they had signs along the sidewalk providing the wait time estimate). So I left.
Today, different strategy – I got there at 10:30. I’ll get there BEFORE the noon rush, says clever me. It was worse — a two and a HALF hour wait. So I gave up again. Guess I’ll try getting there at 6:45 am tomorrow.
I have never seen anything like this in my life. I have no idea what to make of this – especially with apparently unreliable polls…….whole lotta white seniors in the lines….but what does THAT mean? Obama has been trying to scare Florida seniors that Lucifer Ryan and Bain Romney are gonna take away their medicare. And none of the old folks at my Mom’s Asst Living Facility have seen any Libya news, since the (mostly African American) staff only has CNN and Turner Movie Classics on the common area t.v.’s…….
But on the other hand, my colleague who is Jewish, says her parents over in Ft. Lauderdale are noticiing what they think is a subtle and quiet but distinct shifting within the (very numerous) Jewish voting block; they are convinced that Obama will not get anywhere CLOSE to the 78% Jewish vote this time around….maybe as low as 60-40.
And for the first time in my life, (no doubt due to that HHS mandate) our Catholic Diocese has been pushing HARD on the issues, and telling those of us in the pews — well, essentially that we are to place our Catholicism ahead of party affiliation, with some not-so subtle hints as to what that means from a morality in the voting booth standpoint.
So I will be STUNNED if Obama gets a 53-47 advantage among Catholics again. I think 50-52%+ for Romney could be a possibilty, which would represent a complete reversal….
All I know is that I am sick of the daily polls; and I can’t wait for this to be over; for better or for worse. Please Lord, no repeat of 2000, with its agonizing uncertainty and delay!
Watch for those polls to change even further in Romney’s favor. Two-thirds of jobs go to immigrants during Obama’s four years
“Two-thirds of those who have found employment under President Obama are immigrants, both legal and illegal, according to an analysis that suggests immigration has soaked up a large portion of what little job growth there has been over the past three years.
The Center for Immigration Studies [an independent, non-partisan, non-profit, research organization] is releasing the study Thursday morning, a day ahead of the final Labor Department unemployment report of the campaign season, which is expected to show a sluggish job market more than three years into the economic recovery.”
Romney or someone needs to publicize this study and quickly. Explains a lot about the job market doesn’t it? No good paying jobs and the lesser paying ones are going to the illegals…and mobile legals who haven’t established roots.
First Benghazi and now this, it’s a tsunami of bad news for Obama. One might even suspect that “Those whom the gods wish to destroy they first make mad” (prideful, arrogant, narcissistic and criminally foolish)
Possibly we’ve already discussed the following here on this forum . . . but here I go . . .
It’s cool to be pro-Obama. Many people’s self-image is irrevocably tied up in being cool –and “progressive” and “compassionate”, etc., but I’m discussing only the “cool” aspect right now.
For those who are not as leftie (“progressive” and “compassionate”) as they are “cool”, some will of course tell friends and acquaintances that they’re supporting Obama, and they’ll even tell a pollster that they’re supporting Obama.
But, especially if they have friends or acquainances who are out of a job, or especially if they are dismayed at Obama’s apologies for America and/or how he’s lied about Benghazi, . . .
What will they actually end up doing in the privacy of the voting booth?
(Of course, you know what ^I^’m hoping!)
Normally, national polls are more accurate. What did the early voting polls say?
Do people lie to pollsters; mess with their minds?
Having reviewed hundreds of GOP campaign survey’s from the top dozen GOP research firms over the past 25 years, I’ll take a shot at answering a couple of your questions about survey methodology:
– there is no difference between state and national methodology within a partisan firm
– all reputable firms are using live interviews, some “shoe-string budget” lower level campaigns can only afford “robo-polls” but no reputable statewide or national campaign will base a 7 figure messaging budget on a “robo-poll.”
– both state and national polls are conducted at the same time of day
– cell phone user sampling is problematic – some states have very restrictive laws surrounding unsolicited calls to cell phones that do not apply to land lines
Your point about the challenges of getting reliable samples is one the industry recognizes. however I’m actually surprised at the homogeneity of at least the national surveys. Today’s RCP average appears to be based on 10 national survey’s 7 of the ten have it either a 1 point race or dead even and an 8th has Romney +2. The other two feature Romney +5 and Obama +5.
Don’t forget about margin of error (MOE) – it applies to each number – that is a 47-47 race with a 3 point MOE could be 50-44 EITHER WAY . . . i.e. a Romney +5 Gallup survey is the same as an Obama +5 National Journal survey when MOE is accounted for.
It’s important to note that all surveys balance their sample and make assumptions about the partisan composition of the electorate – there’s no industry handbook on either and there is a large degree of variance between campaign and media pollsters on both those issues.
As for the relative “bounciness” of the PUBLIC state survey’s – they generally have a larger MOE than national survey’s and you get more “in-state” pollsters who wouldn’t touch a national survey . . . sampling challenges get magnified, sometimes for better, sometimes for worse.
All in all, no one really knows what’s going to happen. That there are limits to what we can glean from survey’s is summed up best by PEW’s look at early voting so far . . . they report that Romney leads 50-43 among the 19% who have already voted. So 7% of folks who have already voted either can’t or won’t tell us who they voted for.
Typo on the PEW number – it’s 50-43 Romney. But of course those sub-samples have a very small sample size and thus higher margin of error.
Margins of error are frequently misunderstood.
The usual (industry standard) margin of error generally is 95 percent confidence. What does that actually ^mean^??
Say it’s Romney 47 percent, plus or minus 3 percent.
Then — assuming the sample is sound, a separate issue — in the long run (for which we have neither time nor resources) the mean of 95 percent of all such samples will fall between 44 percent and 50 percent for Romney.
Note that 5 percent of all such samples, or 1 out of 20 in the long run, will have a mean that falls outside of 44 percent to 50 percent.
Also, there’s nothing inherently sacred about a 95 percent confidence interval. Use a 90 percent confidence interval, play the same game, and you’ll get a tighter margin of error — now it’ll be (say) 45 percent to 49 percent (I’m fabricating this to keep it simple, but the interval ^will^ be smaller with less confidence) — but now, 10 percent of all such samples (1 in 10) will have a mean that falls outside the confidence interval.
Want more confidence? We can oblige, but at the cost of a wider confidence interval. I’ll fabricate another number here to keep keeping it simple. We can go for 98 percent confidence, but now the margin of error will be 4 percent (fabricated, except we know the interval will now be more generous), or an interval of 43 percent to 51 percent. But now only 2 percent (1 in 50) of all such samples will have a mean that falls outside the confidence interval.
That’s how statistics works: plug the dam here, but you gotta give a little over there. Very squishy, but that’s life (and statistics).
Are you kidding this is fun!!! I used to live in CO and now in another swing state. We’ve forwarded our old CO phone number to a cell phone that rings constantly with pollsters.
The new phone also rings constantly with pollsters.
I’m having a ball!! I don’t tell anyone what I’m thinking and I’ve all the folks at work now doing the same thing. We’ve been asking others if they’re answers are correct and, of those that answer polls, almost all say their gaming the system.
Long story short. BHO’s numbers are being over reported due to folks that have no intention of voting for the current president.
‘the mean of 95 percent of all such samples will fall between 44 percent and 50 percent for Romney’
To get zero margin of error just poll everyone in the country.
Or why not just give the p value and be done with it? That’s the way science is usually done. Statistics is simply an estimate of whether or not the mean of two sets of numbers are truly different. Another way of saying this is that it is an estimate of how closely your sample represents the actual population.
I wasn’t kidding about my question on whether people lie to the pollsters. Sociology is not a science. All these types of sociological surveys depend upon a number of uncontrollable factors including the type of sample. For example most people will not answer and that is an immediate red flag. Do people who refuse to answer prefer Romney or Obama?
What about people who only own cell phones? Do they vote disproportionately for one candidate? What about people who lie or who change their minds at the last minute? Back in the days when I actually voted I did that once.
I think polls really are for the dogs.
Most people glaze over when math or stats are involved; no one (except for us geeks) are in a position, practically speaking, to understand a p-value and what assumptions and analysis go into it.
It got to where ‘they” had to at least offer a margin of error to go with the mean, but who (except M J R and Steve D) knows what a margin of error is all about.?
(Being facetious: quite a few others of us out there know as well, but how many use the term without understanding what it does and doesn’t mean? I figured I’d try to at least offer an idea to those willing to listen/read.)
But Steve D, thanks for your counter-comment!
A good question might be, as the number of people responding to these polls has shrunk, has one side’s participation shrunk more?
That is, are Republicans less likely to answer a poll than Democrats? Personally, I think this is true. IN the same vein that displaying a Romney bumper sticker or yard sign has a decent probability of ending in vandalism or saying anything pro-Romney in a crowd will likely incite a shrieking mob, Republicans are now wary of answering polls. I think there’s a fear that not only are the polls working for liberal organizations, your name would be put on a “list” somewhere for answering incorrectly to the poll (Romney).
Steve D Says at 2:51 pm:
“Do people lie to pollsters; mess with their minds?”
Yes, I think they do.
Who likes to be called at 7:30 pm after a hard day at work and asked a bunch of loaded questions by someone who won’t tell you whose poll it’s for?
And don’t forget the Bradley Effect.
‘Yes, I think they do.’
So, I wonder if anyone has studied this – looked for systemic bias in the polls. On the other hand if both sides mislead equally then it might be hard to detect.
‘but how many use the term without understanding what it does and doesn’t mean?’
I’m a scientist (biochemist) and I can say that unfortunately there are many scientists who do not really understand statistics either even though they use them to make decisions. My main concern though is that there are so many bias, counter effects and complexities to make polling a very hazardous occupation. Still, it seems to me that over the years the polls tend to trend towards overestimating the liberal support and underestimating conservative support, probably for some of the reasons discussed in the thread. Does anyone have any evidence of this, or the contrary?
To stir the pot even more:
http://www.breitbart.com/Big-Government/2012/11/01/Eight-reasons-Pro-Obama-Polls-Are-Wrong
Here’s another thing to consider about those so-called undecided in the polls. Take the case of a person who knows he will not vote for Obama but has not yet decided which of the other five candidates he supports. Technically he’s in the ‘undecided’ category but of course he would never swing to Obama and so his vote (and contribution to the poll) is forever closed to Obama.
‘To stir the pot even more’
No matter which side wins, the business of making polls loses.
I prefer to flip a coin.
Don’t forget the polls were predicting dems would hold the House in the 2010 midterms. They had a D+4 bias. I think the actual numbers turned out to be R+4/5. I think dem pollsters now (most showing Obama ahead) have a bias of D+4. As neo says: garbage in, garbage out.
How does the sail become the anchor?
The wind does not blow.
How are the people endangered?
Information does not flow.
On the other hand
God did not make man
Subject to senses
Lacking of the plan.
Even Nate Silver is admitting that there’s a one-in-six chance that the state polls, with their pitifully small response rates and wide margins of error are categorically wrong for the above reasons.
16% is a huge hedge, because if the state polls are wrong, then Romney wins in a massive landslide and easily takes 350 EVs.
One obvious problem with this “discrepancy” story – there is no discrepancy.
The RCP average of national polls has Obama ahead – barely, but he is ahead. Even Rasmussen has the race tied.
Besides, why would anyone find it strange that Romney could be ahead nationally, but behind in the electoral count? Run up huge margins in solid red states and your popular vote is high, but that doesn’t mean you get to 270.
@cpins
“It’s important to note that all surveys balance their sample and make assumptions about the partisan composition of the electorate ”
This is not true. Most firms will weight their raw results to insure that their sample is representative relative to certain near-constants, such as sex, race, and class ratios. But not partisan leanings. Party ID is almost as much of a variable as is candidate preference. Most reputable firms do NOT adust their raw data to that. Party ID is something to be polled for, not assumed.
One of the clearest examples of this are all the polls that show “independents” rather strongly for Romney, but overall, the two candidates are tied. Plus they often find fewer people identifying as Republicans than expected. Clearly some number of Republicans are self-identifying as independents.
Da, Comrade Joe.
There are no real Republicans or Independents, only lying capitalist running dogs who try to undermine Dear Leader Obama and his Glorious Revolution with their hoarding and wrecking.
Do not worry, though, because as Comrade Maher reminds us, “Black People Know Who You Are and They Will Come After You” if you do not vote for The One.