Home » Did the national polls get it so very wrong?

Comments

Did the national polls get it so very wrong? — 32 Comments

  1. In a prior lifetime I did a lot of pretty high level statistical analysis, which is why I don’t think the earth is warming and why people don’t understand polls.

    Neo is right that for a close election (and this one was close) it is the individual state results are. These are expensive to do compared to national polls and more difficult to compile on a national basis. Had a pollster decided early on to concentrate on swing states and not spend effort at all on all states or national polls they might have done better. There is a non-randomness in small changes in key races that is not easy to model (and probably not meaningful in politics).

    Trump won simply because of the late day disclosures on Clintons email etc. Kellyanne Conway somehow kept Donalds mouth shut during that time. Hillary simply got the bottom of the deck in the last days.

    Right now Donald is looking better (except for a few wild tweets) because it’s quite and people are hoping for normal people selected for cabinet positions. Soon he will have to speak on many issues as a manager not a candidate. Watch the polls.

  2. By the day of the election, the national averages were within spitting distance of being “accurate”, though I still question using error bars that aren’t recalculated based on larger sample sizes. But did the electorate really change their minds in the last two weeks that much? I doubt it- you had an A+ poll swing from Clinton +12 to tied in a week’s period before swinging back to Clinton +5.

    And the state polls in the what were identified as swing states did indeed underestimate Trump’s chances across the board. Also, state polling isn’t harder to do than national polling- that is a canard. It no more difficult to call up 3000 people nationwide and get a response level than it is to call up 3000 people in Wisconsin and get a response level.

  3. Yancey Ward:

    I don’t have time to look it up now, but there are many reasons state polling is harder. Some have to do with sampling, if I recall correctly, but one of the biggees is that there are many states and it’s too expensive to keep taking polls often enough.

    And yes, in this election, people changed their minds a lot. And many didn’t make up their minds till the end. Neither candidate ever polled anywhere near 50%. A lot of wiggle room.

  4. Neo,

    The only objection that makes sense is the one about cost, but that is only true if you try to do national and state polls together. One could have jettisoned all the national polls taken every single week and invested all the funds in polling the following states that one or both camps spent significant time in:

    FL, NC, IA, OH, MI, NH, NV, and PA.

    Do that polling well, and you could have predicted the winner easily. It is a mystery to me why national polling gets the lion’s share of presidential polling dollars- as this election and the one in 2000 shows, it is the electoral election that matters all of the time.

  5. “Trump won simply because of the late day disclosures on Clintons email etc. Kellyanne Conway somehow kept Donalds mouth shut during that time. Hillary simply got the bottom of the deck in the last days.”

    Would add obamacare – renewal notices just coming out with extraordinarily large increases.

    That probably hits at many of the voters in the rust belt that swung trump’s way.

    The absence of discussion of this issue, in general, in the MSM, is remarkable.

  6. Humbug.

    I keep seeing this argument from the intelligentsia.

    Here is the rub Neo, do that same analysis with the polls one week earlier, or one month earlier.

    Think those showing Clinton up 11 a week or two before the election were accurate? They were breathlessly reported as such.

    My Boss predicted this, and I lost $50. He said that the week of the election all the polls would suddenly get “right”, because that last poll is the only one by which they will be judged. He nailed it, and made me look like an idiot.

    What would the final vote tallies have looked like had all the polls except the LA Times, been as accurate as their last?

    The media are not just biased in their coverage, and the polls are a large part of why people are reacting negatively to the media. It is something they have seen over time and thru the years.

    However, if even neo refuses to call it out, and gives them a pass, then they will continue their battle space prep in future elections.

  7. What kinds of polls do you think the campaign’s spend money on? They spend all of it on state level polls, not national ones- they just don’t publicize the results.

  8. And it is a lesson I keep needing to relearn myself- I questioned Trump’s focus on Michigan and Pennsylvania in preference to Virginia, but they clearly had much better data than any public polls of those three states, even though Trump also ran closer in VA than any public poll showed, but he lost by enough of a margin that I can’t question his lack of focus on it now, at least not too much.

  9. What Jim said.

    Even if the final week’s national polling average was fairly close to the popular vote result, the same cannot be said for a good number of the national polls that were published earlier, which tended to show a comfortable (if not large) lead for Clinton throughout the months of September and October. In particular, I remember the ABC tracking poll moving from something like Clinton+10 to Trump+1 within the space of a single week shortly before election day, which is too large a swing to be credible. So, even if the national polling was, on average, somewhat accurate on the eve of election day, the same cannot be said for much of what was published before that time, raising the question of pollster shenanigans.

  10. “Even if the final week’s national polling average was fairly close to the popular vote result, the same cannot be said for a good number of the national polls that were published earlier, which tended to show a comfortable (if not large) lead for Clinton” – not Neo and Jim

    So, y’all don’t think the Comey announcement and the obamacare notices in the final week or so had ANY impact whatsoever?

    With millions fewer voting (as it turns out), historically high negatives for presidential candidates (#1 and #2 worst), and historically high undecideds all along, there was bound to be large swings.

  11. So, y’all don’t think the Comey announcement and the obamacare notices in the final week or so had ANY impact whatsoever?

    Maybe a percentage point or two, which is significant but not an earth-shattering 11-point swing significant. The large majority of people have made up their mind by the final week.

  12. Polling CAN NOT work. Statistical analysis is only valid for relatively stable systems with many actual outcomes to validate the model.

    Nate Silver made his bones predicting baseball games. You have 30 teams playing the same 13 players in the same ballparks and they play 2,340 games a year. Lots of outcomes to test the predictions. Even then there is an inherent 6% error rate according to the 538 website.

    Elections are a whole ‘nother animal. Each election has a different electorate – every year 2.6 million eligible voters die and 2.6 million come of voting age. The candidates are different. The issues are different. The world is different. And you only have one result.

    The polls are a SWAG – a Scientific Wild Ass Guess.

  13. Big Maq, 5:31 pm — “So, y’all don’t think the Comey announcement and the obamacare notices in the final week or so had ANY impact whatsoever?”

    Meh, 5:47 pm — “Maybe a percentage point or two, which is significant but not an earth-shattering 11-point swing significant. The large majority of people have made up their mind by the final week.”

    And to confound the analysis, my understanding is that an unusually large number of people voted early this year, maybe in the neighborhood of 20 percent (?), which introduces something of a dent into any Comey late announcement effect.

  14. The real bias inherent in national polling is the fact there are 50 elections for President. National polls just set up the electorate for disappointment when the popular vote doesn’t line up with the results of the 50 state elections.

  15. Jim Doherty; Meh; M J R:

    It’s never good to try to reconstruct these things without looking them up and being aware of the total picture, rather than picking and choosing your memory of a certain poll or two.

    First of all—yes, there were swings this year, in part because neither candidate was liked and there was a fairly large undecided group compared to previous years. So the news du jour could have quite an effect.

    And yet the swings were not nearly as big, on average, as what you might think you remember. Here’s Nate Silver, writing three weeks before the election:

    Hillary Clinton has a significant lead, although there’s some question about the margin. For instance, one major national poll released on Sunday morning, from ABC News and the Washington Post, had Clinton ahead by 4 percentage points. Another, from NBC News and the Wall Street Journal, had Clinton up 11 points instead. Our forecast model falls in the middle and shows Clinton with a 6- or 7-point lead.

    So, it wasn’t such a large swing after all, measured overall, from that high point until right before the election.

    More from Silver, 3 weeks prior to the election [emphasis mine]:

    Obviously, we’re getting closer and closer to Election Day, with early voting already underway in many states. But the number of undecided voters remains fairly high (although it’s declined slightly). In national polls, about 85 percent of the vote is committed to Clinton or Trump, as compared with around 95 percent that was committed to President Obama and Mitt Romney at this point in the campaign four years ago. Those unpredictable undecided and third-party voters are why our models show both a better chance of a Trump victory than most of our competitors…

    …As my editor put it, everything is on the table in terms of how the final three weeks could go…

    After polls in mid-September showed a very close race, with Trump trailing Clinton by only 1 to 2 percentage points nationally and almost catching up to her in the Electoral College, Clinton began to pull away following her performance in the first presidential debate on Sept. 26. And her lead has continued to grow from about 3 or 4 points just after the debate to more like 6 or 7 points now.

    So it had been a close race for a long time, and then it crept up, and about three weeks before the election it reached what turned out to be the highwater mark for Clinton, a 6-7 point lead. Then it started to close again pretty quickly, until right before the election it had reached the levels I wrote about in the post.

    That Nate Silver piece, three weeks before the election, was written on Oct. 16. The Billy Bush tapes that caused a backlash against Trump had been released on Oct. 8. By voting day, the polls had closed again to a very small margin between the two candidates. I see no reason to believe that the swings that were measured in the polls were not accurate. They seem very reasonable and quite accurate to me.

  16. Jim Doherty:

    Take a good long look at my comment right above this one.

    But in addition, I’d like to say the following to you—just because the “intelligentsia” says something, it doesn’t mean it’s not true.

    Also, you write:

    However, if even neo refuses to call it out, and gives them a pass, then they will continue their battle space prep in future elections.

    I assume that if you have read my comment above this one, you are aware of why I “refuse” to “call it out.” But let me assure you—although I have no idea why I would even need to assure you of this—I have no particular interest in defending pollsters. I am not a pollster. None of my relatives or friends are pollsters. I don’t even play a pollster on TV. I look at the data, study the arguments pro and con, and come to the best conclusions I can, based on that. I have no particular agenda. I call them (or call them out) as I see them.

  17. Just before the election, I thought Mr. Trump was more likely than not to win, but I had to be realistic in assuming it could have gone the other way. It’s nice to be able to watch TV now without getting bombarded by political ads, since I live near a swing state.

    Voter turn-out for 2016 will probably have been around 55% of eligible voters, even with the first woman and the first non-politician candidates. And while Mrs. Clinton got more popular votes, such a result is not as controversial as it was in 2000, with the margin so very close in that one state of Florida.

    The state-by-state results are very interesting, as it is harder to see how a more conventional Republican candidate could have won the same states that were the key to Trump’s electoral victory, without the same appeal from his personality and his policies.

    One also wonders if Trump could have won the states of Minnesota, Nevada, and New Hampshire, if there had not been the same third-party candidates (and if Trump had less personal flaws). Johnson and Stein both received many more votes than in 2012.

    Again from the state results, Mr. Trump had the broad appeal that was needed to win (with 30 states and over 300 electoral votes). He also had multiple avenues to success, even with all the votes still uncounted, which was why Mrs. Clinton conceded early on.

    Maine even awarded Mr. Trump one electoral vote, from its second district, which went 52% to 41% for Trump (nearly the opposite of the first district, which went 54% to 39% for Clinton; there really are two different Maines). The results in Massachusetts were pathetic, 60.8% to 33.5% for Clinton, which leads one to wonder:

    Just who did Neo-Neocon vote for, anyway?

  18. Neo,

    In reality, Clinton never had a 6/7 point lead in October–Nate and the pollsters were wrong. Think about it: if that were true, then Clinton should’ve won the popular vote by something like 4/5 points, as Trump cleaned up among the 15% that were still undecided in October by 10-15 points (doubtless the FBI controversy was a factor here). So, the math suggests that the race was always tighter than the October polls were (on average) suggesting.

  19. And yet the swings were not nearly as big, on average, as what you might think you remember. [Neo]

    The final RCP average was Clinton +3.2, as of October 17 the average was Clinton +7, which amounts to nearly a 4 point shift in the national polling average within the space of the final two weeks of the campaign when less than 15% were still undecided (per Nate). That is too big a shift at the very end of the campaign to be credible, as it would translate to Trump winning undecideds by more than 25 points! Trump did well among late-breaking undecideds but not that well.

  20. Nate Silver simply assigns a percent chance of winning to the established favorite. He rarely if ever takes a contrarian viewpoint. Look at his NFL picks from this week. Every single one was the Vegas favorite, and the one slight upset predicted (the LA Rams) inevitably lost. His sure-fire lock of the week, with a robust 84% chance of winning? The KC Chiefs, who also lost.

    There were only three surprises in this presidential election–Pennsylvania, Michigan, and Wisconsin. Silver (and the rest of the pollsters) whiffed on every one of these states and thus predicted Hillary as the winner. If he called just one of those states for Trump, he would have stood out from the crowd of pollsters who said exactly the same thing as him.

    In 2008, all Silver did was predict that every state that had a chance to vote blue would. You had a popular candidate, the assurance of a white guilt vote, and an uninspiring old moron as opposition in McCain. It was like shooting fish in a barrel. But the Left is naturally suspicious, they project their own voter fraud schemes onto the Right, they worried about some “vast right-wing conspiracy” to steal the election. So when one of their own (Silver) had the “courage” to predict the election landslide that all of us on the right also precisely saw coming, he was hailed as a “genius”. He has since gone on to make no predictions of any significance, save for Obama beating Romney, and the two times the pre-tournament favorite went on to win the NCAAs in basketball (UNC in 2009, Louisville in 2013).

    It’s a tough job, but polling just comes down to people reassuring the public how comfortably the expected winner is going to win by, backed up by math or no.

  21. @Meh – Ya wanna make it as something that is only based on your “feeling” rather than getting back to the measurable whys of your argument.

    You linked to RCP’s national average. Fine, even though when one digs into it, we really ought to be looking at state polls as more relevant to the ecv win. But please walk us through how you get to your conclusion from there.

    Also, examine the “margin of error”, as one factor. You haven’t shown why that doesn’t explain it.

  22. Big Maq,

    This isn’t about “feelings,” I’m just teasing out the implications of how the 4 point shift seen in the national polling average from mid-October to just before election day is not consistent with how undecideds actually broke for Trump during that time. My conclusion is that even if it is granted that the final national polling average as of November was roughly accurate, the same cannot be said for October.

    However, I remain unconvinced that even the final week’s national polling average was accurate. If Greg Phillips is correct in saying that over three million non-citizens participated in the 2016 election, then Trump would have almost certainly won the popular vote if only eligible citizens had voted, and the final national polling average would’ve been off by roughly 3 points (assuming the illegal vote goes overwhelmingly for Clinton).

  23. If what you guys are saying, neo included, were true, then sometimes those polls would show the republican in the lead and he would under perform on election day. But its such a coinky dink that its always dem’s who are shocked. I grant we went thru a spell when we bitched about the weighting etc.

    Still. Its always the dems who are over sampled, remember how the exit polls showed bush losing, and the shock when he won reelection?

    I am not a conspiracy crank. But how many times do we have to see the media pretend to be subtle, when they are not?

  24. Jim Doherty:

    As I’ve said before, there are reasons there are trends in polls, and those reasons need have nothing to do with purposeful bias. For example, basing turnout on a previous election is a typical one.

    Plus, there were times when Trump was ahead in the polls, as you would see if you actually look at timelines of the changes in the polls. What’s more, various surveys (LA Times in particular) almost always showed Trump in front. That was because they used a different polling method.

    As I’ve also said, purposely and falsely putting Clinton ahead could have backfired on her just as easily as it could have helped her. So it makes no sense even conceptually.

  25. Meh:

    Silver’s late average was considerably lower than that, as this article makes clear. If I recall correctly, the RCP average averages in too many earlier polls from that Billy Bush week when Clinton was uncharacteristically higher, and gives a false picture of what’s happening towards the end.

    And let’s have some actual figures for how those “undecideds” broke:

    Except that’s not the whole picture. Some of the polls were wrong to a degree, yes, but there was also something at work in the final days of the election: People who decided late broke strongly for Donald Trump in the states that mattered, according to exit polls. And without this apparent late surge, Hillary Clinton would be our president-elect – not Trump.

    In fact, if you look at the four closest states where Clinton lost – or, in the case of Michigan, where she’s expected to lose – exit polls show late-deciding voters in each of them went strongly for Trump in the final days. In Florida and Pennsylvania, late-deciders favored Trump by 17 points. In Michigan, they went for Trump by 11 points. In Wisconsin, they broke for Trump by a whopping 29 points, 59-30…

    And these weren’t small groups of voters. The number of undecided and third-party-supporting voters who were still free agents in the final week was as many as 1 in 8 voters nationally — an uncharacteristically high number for the eve of an election. (As Nate Silver noted, it was just 3 percent in 2012.)

    In Florida, 11 percent said they decided in the final week. In Pennsylvania, it was 15 percent. And in Michigan and Wisconsin – states where Trump made a late push – fully 20 percent of voters said they arrived at their choice in the last seven days.

    That’s a lot of people breaking for Trump, and in close states it turned the tide.

    One more thing—for your 10:30 AM comment to be correct, one would have to assume that only the previously self-labeled “undecideds” can change their minds. That is a false premise.

  26. Neo,

    That’s a lot of people breaking for Trump, and in close states it turned the tide.

    Right. Nationwide Trump wins the 15% who were undecided in October by about 15 points or so, which comes to a 2 point national shift in his favor, with the margin of victory varying from state to state. However, if the polls in October were right and Clinton had a 6/7 point lead coming into the final leg of the campaign, then she should’ve won the popular vote by about 4/5 points. This is why so many people were confident that Clinton would win.

    One more thing–for your 10:30 AM comment to be correct, one would have to assume that only the previously self-labeled “undecideds” can change their minds. That is a false premise.

    Sorry, but the others were pretty well locked in. Apart from Clinton having a seizure during a campaign event, or being publicly incarcerated by the FBI/NYPD, the people who were committed to voting for her in late October were not about to change their mind.

  27. Meh:

    They were right 3 weeks prior to the election. Then the polls changed as the weeks passed and the election got closer, and they changed not just because the undecideds changed their minds or made up their minds. There were still plenty of undecideds right before the election, and the polls had already changed.

    Other people, who felt they had decided, changed their minds, and kept changing their minds. I know such people. We don’t know how many there were.

    Nor were the polls perfect. But they were pretty darn good.

  28. Neo,

    They were right 3 weeks prior to the election.

    No they weren’t! As of October 18, the RCP national polling average was Clinton +7.1. If that was right, then she should’ve won unless Trump wins undecideds by something like 30 points nationwide, which wasn’t going to happen.

    Look, we can also get into the specifics of how certain polls were rather drastically “re-weighted” from D +9 to more reasonable samples as election day got closer. It’s much easier to say that the polls were off in October, but were more reasonable in November (though still too favorable to Clinton).

  29. Meh:

    You keep ignoring my main point, which I’ve made over and over, about a fallacy in your reasoning about which voters might be changing their minds.

    Hint, hint: it doesn’t have to just be people who call themselves “undecided.”

    It’s a waste of time to talk to you further about it. Go back and study what I’ve already written.

  30. Polls are designed to manipulate public opinion, not reflect it. Once people start trying to parse out if it is true or false, the trap activates.

  31. @Neo – agree it is worthless carrying on.

    Once people have in their mind that everything is essentially “rigged”, then they will always have a story for why that is so.

    Might as well be arguing over how a thermometer is biased because the manufacturer, the weatherman, … everyone! …, is aligned to deceive us into believing in CAGW.

    There is no basis to believe in any objective observation, especially when it is based on the black arts of statistical analysis.

Leave a Reply

Your email address will not be published. Required fields are marked *

HTML tags allowed in your comment: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>