Advertisement

SKIP ADVERTISEMENT

What Went Wrong With Polling? Some Early Theories

Pollsters thought they had learned from the errors of 2016. It’s possible that they did, and that this election reflects new problems.

Asking for a polling post-mortem at this stage is a little bit like asking a coroner for the cause of death while the body is still at the crime scene. You’re going to have to wait to conduct a full autopsy.

But make no mistake: It’s not too early to say that the polls’ systematic understatement of President Trump’s support was very similar to the polling misfire of four years ago, and might have exceeded it.

For now, there is no easy excuse. After 2016, pollsters arrived at plausible explanations for why surveys had systematically underestimated Mr. Trump in the battleground states. One was that state polls didn’t properly weight respondents without a college degree. Another was that there were factors beyond the scope of polling, like the large number of undecided voters who appeared to break sharply to Mr. Trump in the final stretch.

This year, there seemed to be less cause for concern: In 2020, most state polls weighted by education, and there were far fewer undecided voters.

But in the end, the polling error in states was virtually identical to the miss from 2016, despite the steps taken to fix things. The Upshot’s handy “If the polls were as wrong as they were in 2016” chart turned out to be more useful than expected, and it nailed Joe Biden’s one-point-or-less leads in Pennsylvania, Georgia and Arizona.

The polls were off in 2020 in almost the same ways they were off in 2016.

Final 2020
poll avg.
2020 polls with
2016 error
2020
Result
U.S. † +8 Biden +6 Biden +5 Biden
N.H. +11 Biden +7 Biden +7 Biden
Wis. +10 Biden +4 Biden <1 Biden
Minn. +10 Biden +4 Biden +7 Biden
Mich. +8 Biden +4 Biden +3 Biden
Nev. +6 Biden +8 Biden +3 Biden
Pa. † +5 Biden <1 Biden +1 Biden
Neb. 2* +5 Biden +9 Biden +7 Biden
Maine 2* +3 Biden +9 Trump +7 Trump
Ariz. +3 Biden +1 Biden <1 Biden
Fla. +2 Biden <1 Biden +3 Trump
N.C. +2 Biden +3 Trump +1 Trump
Ga. +2 Biden <1 Biden <1 Biden
Ohio <1 Trump +6 Trump +8 Trump
Iowa +1 Trump +5 Trump +8 Trump
Texas +2 Trump +4 Trump +6 Trump

† These reflect Times estimates of the final vote margin once all votes are counted. * In Maine and Nebraska, two electoral votes are apportioned to the winner of the state popular vote, and the rest of the votes are given to the winner of the popular vote in each congressional district. (Maine has two congressional districts, and Nebraska has three.) Poll error in 2016 is calculated using averages of state polls conducted within one week of Election Day.

The national polls were even worse than they were four years ago, when the industry’s most highly respected and rigorous survey houses generally found Hillary Clinton leading by four points or less — close to her 2.1-point popular-vote victory. This year, Mr. Biden is on track to win the national vote by around five percentage points; no major national live-interview telephone survey showed him leading by less than eight percentage points over the final month of the race.

The New York Times/Siena College polls were also less accurate than they were in 2018 or four years ago. In 2016, the last two Times/Siena polls were among a very small group of polls to show Mr. Trump tied or ahead in Florida and North Carolina. This time, nearly all of the Times/Siena surveys overestimated Mr. Biden to about the same extent as other surveys.

In the months ahead, troves of data will help add context to exactly what happened in this election, like final turnout data, the results by precinct, and updated records of which voters turned out or stayed home. All of this data can be appended to our polling, to nail down where the polls were off most and help point toward why. But for now, it’s still too soon for a confident answer.

In the broadest sense, there are two ways to interpret the repeat of 2016’s polling error. One is that pollsters were entirely wrong about what happened in 2016. As a result, the steps they took to address it left them no better off. Another is that survey research has gotten even more challenging since 2016, and whatever steps pollsters took to improve after 2016 were canceled out by a new set of problems.

Of these two, the latter interpretation — real improvements canceled out by new challenges — may make the most sense.

“I think our polls would have been even worse this year had we employed a pre-2016 methodology,” said Nick Gourevitch of Global Strategy Group, a Democratic polling firm that took steps to better represent Mr. Trump’s supporters. “These things helped make our data more conservative, though clearly they were not enough on their own to solve the problem.”

The Daily Poster

Listen to ‘The Daily’: About Those Polls…

Joe Biden may have won the election, but the margin of victory was much closer than the experts predicted. Why?
bars
0:00/35:13
-35:13

transcript

Listen to ‘The Daily’: About Those Polls…

Hosted by Michael Barbaro; produced by Jessica Cheung and Michael Simon Johnson; and edited by Lisa Chow and M.J. Davis Lin.

Joe Biden may have won the election, but the margin of victory was much closer than the experts predicted. Why?

michael barbaro

Hey, it’s Michael. This episode has been updated to clarify our analysis of the Latino vote in the election.

From The New York Times I’m Michael Barbaro. This is “The Daily.”

Today: What the results of the election taught us about the American electorate, and about the polling that once again failed to correctly understand it. A conversation with my colleague Nate Cohn.

It’s Tuesday, November 10.

So Nate, we want to have two conversations with you. The first is about what we learned from the vote, the results of this election, now that the race has been called. And second is what we learned about why the polls were so off once again, including our own polling here at The Times, and kind of have a reckoning on that front.

nate cohn

Sounds good.

michael barbaro

Right, I’m sure it’s going to be a total pleasure for you.

nate cohn

It will be.

michael barbaro

Let’s start with the voters and how this historic turnout for both candidates broke down. And perhaps the most intriguing group in this campaign, and so far in the results, are Latino voters. So tell us what the final-ish data is telling you about Latino voters?

nate cohn

Well, I suspect that many listening right now have probably heard that Donald Trump fared quite well among Latino voters. You learned that pretty quickly on election night when the results from Miami-Dade county in Florida come in. And now that we have seen more detailed results from elsewhere in the country, I think we can safely say that Latino voters really swung to Donald Trump. Estimates vary about Latino support for Trump in 2016, but it’s somewhere in the 25 to 30 percent range.

This year, we are still waiting for all the numbers, but we’re expecting that to look more like 35 percent. And, there’s been important focus this year on the fact that the Latino vote is not a monolith — hence the divided support for Biden and Trump.

But what’s remarkable about this year is that when you look at the results so far, this increase in support for the president is consistent all over the country. It’s true in the agricultural regions of California, like the Imperial Valley. It’s true in the border towns along the Rio Grande. It’s true, as you know, in Cuban areas like in Miami-Dade County. But it’s also in Puerto Rican areas or around Orlando and Kissimmee. It seems like it’s even true in the northern cities like Philadelphia or Milwaukee, where Latino voters are usually the very most reliable for Democrats. And the magnitude of the improvement for the president is really significant. There are counties along the Rio Grande where Trump picked up 50 points.

michael barbaro

50 points among just Latino voters?

nate cohn

Yeah, I mean, you can’t know for sure that there isn’t some contribution of white voters in these counties, of course.

But these are places where Latino voters make up the overwhelming majority of the electorate. We can say the same thing for these Latino enclaves in Philadelphia or in Milwaukee. The magnitude of the gains there is not as staggering as it is among the Rio Grande, but it’s still really stark: 10, 15, 20-point gains for Donald Trump.

michael barbaro

So, Nate, how do you start to explain what is now a much broader phenomenon than perhaps many of us thought it would be on election night? Is this the story of Democrats failing to capture this vote? Or is this the story of Donald Trump succeeding in capturing it? Is it both? How are you thinking about this?

nate cohn

Yeah, the first thing I would just say is I think this is a place where reporting on the ground will be really helpful. The polls struggle with Latino voters. On a national survey, you only get 100 Latino voters. It’s difficult to amass a lot of Latino respondents to do the kind of deep dives that we can do on larger subgroups like white voters, for instance, who I’m sure we’ll get to. As a general framing point, I would say that this election was about Donald Trump. It was a referendum on the president. Joe Biden did everything he could to make sure it was about the president. He stayed in his basement as the Trump campaign often noted. And so I am inclined to assume that when we see big shifts that it reflects attitudes about the president, not Joe Biden.

michael barbaro

So we should read into this that the president successfully appealed over the last year or so to this group of voters?

nate cohn

That’s right. One thing I would point out is that there was always strong evidence that the economy was the president’s strong suit in this election. And when you look at the results, not just for Latino voters, but also for other non-Hispanic voters, it does seem to me that the president made his largest gains in less affluent areas. It would not surprise me if that reflects the greater salience of the economy for economically vulnerable people, compared to people who might have the privilege to vote on other issues like cultural issues or the president’s conduct in office and so on.

A second observation I would make is that to me this was not an election about immigration in the same way that the 2016 election was about immigration. Trump’s pledge to build the wall. His comments on his very first day he sought the presidency when he said that Mexicans were rapists and criminals and so on. Hillary Clinton in 2016 really focused on immigration as one of her major critiques of the president. And I wouldn’t say that was nearly so true in 2020. I don’t think immigration even came up in the first presidential debate. So I think you can imagine that immigration became less salient to not just Hispanic voters, but all voters. But that was particularly consequential among Hispanic voters, because I think we can at least hypothesize that many Latino voters in 2016 were skeptical of the president in part because of the way he talked about this issue.

michael barbaro

So the issue that might have hurt President Trump with Latino voters — immigration — was far less prominent this time than four years ago. And the thing that would potentially help him the most with Latino voters — the economy — did seem to work in his favor, insofar as Latino voters identified him with economic prosperity, especially pre-pandemic economic prosperity?

nate cohn

Yeah, that’s right. And one final point I would make is that the president has always done really well among white voters without a degree. His schtick has had appeal for that demographic group. And I don’t think that’s just about policy. I also think it’s about his conduct. I think that the Trump act has appeal to a lot of people who haven’t previously been terribly receptive to Republican politicians in the past.

And I think that if we’re honest, that it’s not always obvious why the president’s appeal to white working-class voters on the economy, or in terms of his conduct and temperament, wouldn’t also have appeal to working-class voters who are non-white, whether they’re Black or Latino.

So I could cobble together a theory that sort of says that many of the things that appeal to white working-class voters about Donald Trump may appeal to Hispanic voters as well. But in 2016, that was obscured by his position on immigration. And once immigration was taken out of the picture, maybe that gave the president a belated opportunity to make the same sort of gains among working-class Latino voters in 2020 that he was able to make among white working-class, traditionally Democratic voters, in 2016.

michael barbaro

It’s interesting, Nate, that you made that parallel, because as you were describing what’s happening here, it occurred to me that if you are the Democratic party, the last major group of voters that were perhaps taken for granted in the past decade or so were white working-class voters, especially in the Midwest. Democratic leaders thought those voters were unquestioningly theirs. I wonder if that’s the right way of thinking about how the Democratic party saw Latino voters in 2020 — that there was an assumption that they would be with Joe Biden. And they just were not to the same degree.

nate cohn

I think that’s right. I think there were a lot of people in this country who became Democrats during an era when the Democratic Party was the party of working people and the Republicans were the party of the rich and business interests, who over the last four years have come to see the president as a different kind of Republican.

And the president has denied the Democrats some of their traditional advantage among working-class voters of all races as a result. This was not a campaign about privatizing social security. It wasn’t a campaign about the minimum wage and so on. Instead, the Democrats have advanced a sort of idealistic liberal message with obvious resonance at least among college-educated voters. And it didn’t have the same resonance among working-class voters. And Democrats in the past have gotten away with it by falling back on their traditional strengths among these groups, long-standing allegiance that has maybe been eroded now the Republicans have put forward a more populist candidate.

michael barbaro

OK, Nate, what about Black voters? What did we learn about this population from the results of Biden’s victory?

nate cohn

I think that there were a few — I would make a few points about what happened with Black voters. One is that it seems to me that although Black turnout increased, it did not increase the same extent as it increased among non-Black voters. And so as a result, the Black share of the electorate seemed to decline.

michael barbaro

The number of Black Americans who voted did increase. But the number of everybody else increased just a little bit more?

nate cohn

That’s right. So you may recall in 2016, the turnout in places like Milwaukee and Philadelphia and Detroit was down. It was down by so much that Hillary Clinton narrowly lost. In 2020, the turnout was up significantly, enough that if you could go back in time with this Black turnout, Hillary Clinton would have been the president. But yet, the white turnout elsewhere in these states increased by even more, such that if you could go back in time and increase white turnout in 2016 to the same extent, Donald Trump would then come back and be president again.

michael barbaro

Wow, that is actually fascinating. You’re saying if you applied the increase in both the Black and white vote that occurred this year back then, you’d get the same result.

nate cohn

That’s right. In the end, the Black versus white turnout dynamic did not change in a way that would have allowed the Democrats to prevail four years ago.

michael barbaro

And among the Black voters who turned out, what percentage supported Biden versus President Trump? Because I have the sense that the president did somewhat better with Black voters than was expected.

nate cohn

I think that’s right to an extent. I think that this is, again, a case where we’ll need to see the final data before we nail this down. But let’s just say that Biden won roughly 90 percent of the Black vote and that Donald Trump won around 10 percent of the Black vote. Give or take, we’ll see what the final numbers are. That will be better for Donald Trump than in 2016. And if I were forced to explain it, I would chalk up a similar explanation to what I told you with Latino voters or some combination of the president’s economic appeal and so on. It was able to help him peel off just a little bit. But I don’t want to overthink it too much. This is a pretty small shift all things considered. And Democrats continue to command the overwhelming support of Black voters.

michael barbaro

Nate, if your theory is accurate and what connects the improvement by Donald Trump among Latino and Black voters is his economic message, what’s interesting about that, of course, is that the economy is doing terribly right now because of the pandemic. And so it means that there’s a vestigial affection for the president’s success a year ago, the way the stock market did before the pandemic, the way job growth was going before the pandemic. But it’s remarkable that he’s still getting credit for that given where things are right now.

nate cohn

Here are the points I would make. One, you’re totally right. I think that the president does still get credit for the way he handled the economy during ordinary circumstances. And people can rationally believe that the president was a good steward of the economy when coronavirus wasn’t in the picture. And therefore, he’ll be a good source of the economy once coronavirus is out of the picture.

And the second thing I would say is that I think that he also has gotten some credit from people for the way he handled the economy during the coronavirus. The stimulus package was really popular. The Democrats have not criticized the way he’s handled the economy during the coronavirus. They’ve only criticized his handling of the coronavirus. And even on his handling of the coronavirus, the main criticism of him is that he’s too eager to reopen the economy.

So it’s a little bit complicated to make people go through the logic of, here’s someone who wants to reopen the economy. And therefore is on the side of getting people back to work, and so on. But he’s actually bad for the economy because that step that’s facially good for the economy will contribute to the spread of the coronavirus to force shutdowns, which hurt the economy. That’s — I do think that we take that logic for granted in a way that I’m not sure we should expect of ordinary people.

michael barbaro

Fair point. OK, so I now want to turn to a group that we have been talking about a lot throughout this campaign, a group that was predicted to potentially be decisive in this election, and that was suburban voters. Polls predicted that many of the suburban voters who had voted for Trump four years ago would swing to Biden. Did that end up happening?

nate cohn

That’s the thing that ended up happening. And it was just enough for him to get over the top in the northern battleground states as well as in Arizona and in Georgia. Across the country, Joe Biden did better in suburban areas than Hillary Clinton did four years ago. His gains were largest in traditionally Republican suburban areas, like Atlanta, like Dallas, like Jacksonville, Indianapolis, Kansas City, Grand Rapids, Michigan — places that are not full of liberal suburbanites that have been voting Democratic for a really long time, like Westchester County or the suburbs of D.C. and Philadelphia.

And don’t get me wrong, Biden did make gains in a lot of those traditionally Democratic suburban areas. But the biggest gains were in the areas where 10 or 15 years ago, the Republicans dominated among the suburban vote. And now a lot of traditionally Republican, affluent white voters in those areas are saying, wait a second, this isn’t the Republican Party I signed up for. And now it has swung pretty decisively to the Democrats.

michael barbaro

And do we know why there was this shift? It feels like that word you just used, affluence, may be an element of it. These are voters who can financially afford to cast a vote based on whether they like Trump or Biden.

nate cohn

Yeah, I definitely think there’s something to that. I think that there are a lot of rich white people in the suburbs of Dallas and the suburbs of Atlanta, who’ve been voting for their tax cuts for a very long time. But Donald Trump crossed a line with them in terms of his personal conduct. And they are willing to vote on what they feel and think about him as a person in a way that maybe other people are not necessarily so inclined to do.

I also do think there is a policy element of it as well. I think that immigration has always been popular. And free trade has always been popular among a big chunk of conservative voters in suburbs across the South. And the president has departed from traditional Republican views on some of those issues in a way that may at least somewhat complicate the idea that this is strictly a vote against their policy views. But I do think it’s true that it’s a lot easier for someone making $200,000 a year to vote against their perceived economic interests than it would be if you make $20,000 a year.

michael barbaro

OK, so finally, what about white non-college educated voters, many of whom are in rural America, not suburban? So we need to be talking about ex-urban or rural voters. This is traditionally seen as the president’s base. This is the group that Biden had hoped he would start to poach. How successfully did he do that? How well did President Trump defend this base for himself according to the data that you have seen?

nate cohn

Donald Trump defended his base here into a far greater extent than any of the pre-election polls anticipated. As far as I can tell, Joe Biden didn’t make any gains in rural America among white voters since 2016. And as you alluded to, the whole premise of choosing Scranton Joe was that if the Democrats nominated a white moderate with some populist appeal, that you could win back some of the voters who backed Barack Obama in 2012, supported Donald Trump in 2016. And that just didn’t happen. And in many cases, Donald Trump extended his gains. There are places in rural Iowa and rural Ohio, where Donald Trump did even better than he did four years ago. And that this is the exact opposite of what the pre-election polls said. So something went very wrong in the public opinion research there.

michael barbaro

Well, we will get to that. But at the end of the day, if I’m putting all these pieces together the way I think you have intended for me to, it feels like suburban voters end up being the most important group to shift and shift in the direction of Joe Biden. Is that correct?

nate cohn

That’s absolutely right. We have known for a very long time that voters had deep reservations about Donald Trump. We know that he only received 46 percent of the national vote in 2016. We know that he was the least popular candidate in terms of favorability ratings when he was elected. And the president had four years to try and address those shortcomings.

It seems that his handling of the economy and his performance on the job was enough to persuade some number of more economically vulnerable voters, or maybe just voters in general — even higher on the economic spectrum — to support his re-election despite the personal reservations that they’ve probably had about him for years.

But it also seems that in nominating Joe Biden — someone who is fairly broadly appealing and promised to unify the country and to act in a way more befitting of the office — that Joe Biden was able to consolidate a significant number of voters who have had longstanding reservations about the president, especially college-educated voters, especially traditionally Republican tilting independent voters in the suburbs around many of our largest cities. And that, to me, tells you a very odd and interesting story. It says that the economy can really help the president as we’ve already known. But there are limits to it.

michael barbaro

We’ll be right back.

archived recording 1

Let’s take a look at our new poll. And the headline number is this. This is a wow. The largest lead of the race for Joe Biden.

archived recording 2

Joe Biden with a 14-point lead nationally over President Trump.

archived recording 3

10 points.

archived recording 4

12-point lead over Trump.

archived recording 5

The largest lead of the race for Joe Biden by 16-point spread. Joe Biden leads in Michigan. He leads in North Carolina. He leads in Pennsylvania. He leads in Wisconsin.

archived recording 6

Joe Biden is leading President Trump by 11 points in Wisconsin in the latest New York Times Siena College poll.

archived recording 7

Joe Biden is in the hunt in these races that have been going red of late in this state. So imagine Joe Biden’s election night could also include North Carolina, could possibly include Florida. This is not an out-of-the-question map here for Joe Biden, given the state of the race we know.

michael barbaro

So Nate, now for the root canal section of this conversation. You cover polling for The Times. And you are involved in how polling itself is conducted for and by The Times along with our partners. And we all know, at this point, that there was significant polling error this year. How would you describe the level of polling error in 2020?

nate cohn

I would consider this to be far worse than 2016.

michael barbaro

Really?

nate cohn

Far worse. In terms of the difference between the final poll results and the actual results, the difference is not that much worse than 2016. That’s pretty comparable. But pollsters this year do not have the same excuses that they had four years ago. And I think that the error is much more systematic and portrays much more fundamental problems with the effort to represent the electorate than the polling did four years ago.

michael barbaro

And I want you to give us some examples just so we are on the same page. When you say the polls are worse this time than four years ago, what do you mean by that?

nate cohn

Maybe it’s easier to start with 2016 for a second. The polls did a good job of representing white voters without a degree, in that they showed white voters without a degree being great for Trump against Hillary Clinton. But they did not have enough of those voters. And although that is a serious problem in polling — to not have enough of a certain demographic group — it is a common problem in polling. And it is a fixable problem, because you can give more weight to voters from that demographic group. And in the post-election analysis four years ago, pollsters found that when they gave more weight to that group, that the polls did pretty well. In 2020, pollsters did that. They gave more weight to white working-class voters. But they were no closer to the result, even though we know that the same technique four years ago would have brought the polls closer to the result.

michael barbaro

So I just want to be clear on this. You’re saying what makes the error in 2020 greater and more grievous than 2016 is that we entered 2020 having made meaningful adjustments to our polling methodology, all polls, that come to the presidential election. And yet we still had super off polls despite those reforms and adjustments?

nate cohn

Not only were they super off, though, they were just as bad. And what that means is that while four years ago, the polls showed Donald Trump doing really well among white voters without a degree, and had too few of them, this year the polls had the right number of white voters without a degree. But they showed Joe Biden doing way better among this group than he actually did. So for all this election, we’ve been saying white voters without a degree who voted for Donald Trump in 2016 appeared to have come back for Joe Biden. And that didn’t happen.

michael barbaro

OK, so how do you explain the level of error that you just ticked through after all the corrections made four years ago to try to ensure that this would never happen?

nate cohn

I am going to try and answer your question. But before I do that, I do want to say that it is too soon to have a definitive autopsy on this. I’m going to take that analogy a little further. But the body is still dead at the scene of the crime. It has not made it to morgue yet for us to go into the details and dig in to see exactly what went on here. I mean, but we know the polls went really wrong. But it’s going to be a bit until we’ve nailed down exactly why. So I’m happy to speculate.

michael barbaro

But assuming you’re a crime scene investigator and you’re there and you’re trying to figure it out, what do you know so far?

nate cohn

Yeah, I can just toss out some ideas. And that’s all they are at this stage. We have seen over the last four years, a huge increase in political participation on the left. We also know that political participation correlates with participating in polls, which make sense. You get called. And you either choose to take the survey or not. And if that interests you because you’re interested in politics, you are likelier to take a poll. This could mean that Democrats became a little more likely to take a poll. That’s one possibility.

Another possibility is looking on the other side that Trump voters are less likely to respond to surveys than they were four years ago. Perhaps, the president’s attacks on the media and institutions have gradually eroded their trust in surveys. Maybe even the 2016 result itself had a role in diminishing their willingness to participate in surveys, because the polls were off by so much that now they don’t trust them and don’t want to play. We make sure in the state of Pennsylvania that 41 percent of our interviews are with Republicans or whatever the number was in that area. Which Republicans picked up the phone? Were they the ones that were really enthusiastic about the president in rural areas? Or were they the people who are also registered as Republican but are no longer fans of the president?

I think that we have to conclude that we got the ones who are more likely not to be fans of the president. That’s not to say we didn’t get plenty of people who supported the president, but enough of a difference compared to reality that you nudge all of the polls in one direction.

michael barbaro

Well, Nate, we’ve had four years of the president speaking about the organizations that conduct polling in a very specific way. And that would be The New York Times, which conducts polls. That would be ABC News. That would be CNN. That would be CBS. And he has described them as the enemy of the people. He has described them as fake media. So I don’t have a hugely hard time understanding why his supporters would be skeptical of answering our calls.

nate cohn

No, I think that four years of the president advancing that message could have had a negative effect on the polls. I think that’s totally possible. I don’t have any proof of that. But what’s important about that theory is that it could potentially explain why the polls have gotten worse since 2016. That, to me, is the key part of any theory if you really want to try and dig into what’s happening here.

michael barbaro

So, Nate, are we left with the kind of inevitable conclusion here, after two presidential elections that pretty much committed the same sin perhaps for different reasons, that election polling — pre-election polling — is broke?

nate cohn

That’s a great question. And I think it depends on the level of precision that you expect out of it. And there is a spectrum of ways that you can interpret polling errors. One possibility is that the polls are imprecise, but they’re still useful. They’ll never be able to nail down whether Joe Biden is going to win Georgia or lose it by four or win it by four. Like they’re just not that good. But they’re good enough to flag that something happened in Georgia, and that’s useful to us.

Another possibility is that they’re so imprecise that they’re no longer useful. It’s like all right, we got these polls showing Biden up eight points. But in our political era, the range of possible outcomes in national elections is pretty tight. In my lifetime, the whole range of results goes from what? Bush plus two in 2004 to Clinton plus eight in 1996. And so if the amount of error in a poll is basically equal to the range of possible results, they may tell us something. But they’re just not that useful. In any given election, either side can win. And you can’t rule out anything. And you didn’t get that much out of it.

Third possibility is that the polls are so wrong that they’re counterproductive. They’re not just useless. But they give an actively misleading picture of the country that we live in. That would be really bad if true.

michael barbaro

OK, so now I need to know where do you fall on this spectrum? Imprecise but directionally useful; useless but we can live with them; really bad and misleading and counterproductive? Because it feels like for a lot of people, number three is where we may be living right now. Where is your head?

nate cohn

I think there are definitely elements of the polling error this year and in 2016 that fall in to the counterproductive category. In Ohio and Iowa, Joe Biden decided to spend late parts of his campaign, final stop, in Cleveland on the day before the election. That’s outright counterproductive. It’s also counterproductive for readers in the electorate. There are cases where political activists make decisions to spend money on longshot Senate races like South Carolina Senate, or the Kansas Senate race, or the Alaska Senate race where people just put their energies into the wrong spot. There’s also a toll on people’s trust in institutions. Polling is an inherently uncertain thing. And so we do expect to some extent that readers appreciate that polls can be off. And it shouldn’t undermine the credibility of other things that we have to say or do, which could be more based on the firmest facts.

But I think that it does undermine the credibility of our ability to communicate to readers that we understand the difference between something that we know to be true and something that is a best estimate.

michael barbaro

But you didn’t answer my question. Where do you fall?

nate cohn

I’m really — look, I think that —

michael barbaro

It sounds like you’re torn.

nate cohn

I really am torn. I think the alternatives to public polling are pretty bad. I mean, we’re just stuck to talk to our neighbors and our like-minded friends. We wouldn’t really have any idea of what’s happening the rest of the country potentially. So that’s tough to accept. Let’s revisit this after the autopsy. I mean, if we go through the data and we find that there are things that we can fix, then we’ll make those changes and evaluate whether we think we’re still in the imperfect but useful category. And then we might continue to do some amount of public polling in the future.

But if we conclude that we don’t think we can fix these things, then we have a really hard choice, which is whether to abandon the enterprise altogether — which has some important costs, because if you don’t have a read on the attitudes of the electorate, I think we’re left with alternatives that are not very good. And I think it does matter to understand where the American people are at. I think it’s important to the way this democracy works. But if we can’t get there, then we can’t get there. And we would have to re-evaluate what we’ll do going forward.

michael barbaro

Well, Nate, I look forward to hearing what you learn. Thank you very much for your time.

nate cohn

Thanks for having me.

michael barbaro

We’ll be right back.

Here’s what else you need to know today.

archived recording (senator mitch mcconnell)

I want to spend a few minutes this morning talking about what we saw last week, where we are now, and where our great country will go from here.

michael barbaro

In a speech from the capital on Monday, Senate Majority Leader Mitch McConnell threw his support behind President Trump’s refusal to concede the election and declined to recognize President-elect Joe Biden’s victory.

archived recording (senator mitch mcconnell)

President Trump is 100 percent within his rights to look into allegations of irregularities and weigh his legal options.

michael barbaro

McConnell mocked Democrats for calling on Trump to accept the results of the election, saying that many of them had never recognized Trump’s victory four years ago. That message prompted a scolding from Senate Minority Leader Chuck Schumer.

archived recording (senator chuck schumer)

Joe Biden won this election fair and square. Republican leaders must unequivocally condemn the president’s rhetoric and work to ensure the peaceful transfer of power on January 20th. But too many, including the Republican leader, have been silent or sympathetic to the president’s fantasies.

michael barbaro

And the drugmaker Pfizer announced that an early analysis of large-scale human testing has found that its coronavirus vaccine is more than 90 percent effective in preventing infection — a highly promising result that could make it a leading candidate for federal approval. The clinical trial is not complete, and the result could change. But so far, the company said, no serious safety concerns have been observed. Pfizer said they would ask the F.D.A. for emergency authorization for the vaccine later this month.

That’s it for “The Daily.” I’m Michael Barbaro. See you tomorrow.

The explanation for 2016’s polling error, while not necessarily complete or definitive, was not contrived. Many state pollsters badly underrepresented the number of voters without a college degree, who backed Mr. Trump in huge numbers. The pollsters went back to their data after 2016, and found that they would have been much closer to the election result if they had employed the standard education adjustments that national surveys have long used. An Upshot analysis of national surveys found that failing to weight by education cost Mr. Trump about four points in polling support — enough to cover much of the 2016 polling error. Other pollsters had similar findings.

But this time, education weighting didn’t seem to help. State and national polls consistently showed Mr. Biden faring far better than Mrs. Clinton did among white voters without a degree. Last week’s results made it clear that he didn’t.

Over all, the final national surveys in 2020 showed Mr. Trump leading by a margin of 58 percent to 37 percent among white voters without a degree. In 2016, they showed Mr. Trump ahead by far more, 59-30. The results by county suggest that Mr. Biden made few gains at all among white voters without a degree nationwide, and even did worse than Mrs. Clinton’s 2016 showing in many critical states.

In contrast, the 2016 polls did show the decisive and sharp shift among white voters without a degree, but underestimated its effect in many states because they underestimated the size of the group. Many state polls showed college graduates representing half of the likely electorate in 2016, compared with about 35 percent in census estimates.

The poll results among seniors are another symptom of a deeper failure in this year’s polling. Unlike in 2016, surveys consistently showed Mr. Biden winning by comfortable margins among voters 65 and over. The final NBC/WSJ poll showed Mr. Biden up 23 points among the group; the final Times/Siena poll showed him up by 10. In the final account, there will be no reason to believe any of it was real.

This is a deeper kind of error than ones from 2016. It suggests a fundamental mismeasurement of the attitudes of a large demographic group, not just an underestimate of its share of the electorate. Put differently, the underlying raw survey data got worse over the last four years, canceling out the changes that pollsters made to address what went wrong in 2016.

It helps explain why the national surveys were worse than in 2016; they did weight by education four years ago and have made few to no changes since. It also helps explain why the error is so tightly correlated with what happened in 2016: It focuses on the same demographic group, even if the underlying source of the error among the group is quite different.

Polling clearly has some serious challenges. The industry has always relied on statistical adjustments to ensure that each group, like white voters without a degree, represents its proper share of the sample. But this helps only if the respondents you reach are representative of those you don’t. In 2016, they seemed to be representative enough for many purposes. In 2020, they were not.

So how did the polls get worse over the last four years? This is mainly speculation, but consider just a few possibilities:

The president (and the polls) hurt the polls. There was no real indication of a “hidden Trump” vote in 2016. But maybe there was one in 2020. For years, the president attacked the news media and polling, among other institutions. The polls themselves lost quite a bit of credibility in 2016.

It’s hard not to wonder whether the president’s supporters became less likely to respond to surveys as their skepticism of institutions mounted, leaving the polls in a worse spot than they were four years ago.

“We now have to take seriously some version of the Shy Trump hypothesis,” said Patrick Ruffini, a Republican pollster for Echelon Insights. It would be a “problem of the polls simply not reaching large elements of the Trump coalition, which is causing them to underestimate Republicans across the board when he’s on the ballot.”

(This is different from the typical Shy Trump theory that Trump supporters don’t tell pollsters the truth.)

A related possibility: During his term, Mr. Trump might have made gains among the kinds of voters who would be less likely to respond to surveys, and might have lost additional ground among voters who would be more likely to respond to surveys. College education, of course, is only a proxy for the traits that predict whether someone might back Mr. Trump or respond to a poll. There are other proxies as well, like whether you trust your neighbor; volunteer your time; are politically engaged.

Another proxy is turnout: People who vote are likelier to take political surveys. The Times/Siena surveys go to great lengths to reach nonvoters, which was a major reason our surveys were more favorable for the president than others in 2016. In 2020, the nonvoters reached by The Times were generally more favorable for Mr. Biden than those with a track record of turning out in recent elections. It’s possible that, in the end, the final data will suggest that Mr. Trump did a better job of turning out nonvoters who backed him. But it’s also possible that we reached the wrong low-turnout voters.

The resistance hurt the polls. It’s well established that politically engaged voters are likelier to respond to political surveys, and it’s clear that the election of President Trump led to a surge of political engagement on the left. Millions attended the Women’s March or took part in Black Lives Matter protests. Progressive activists donated enormous sums and turned out in record numbers for special elections that would have never earned serious national attention in a different era.

This surge of political participation might have also meant that the resistance became likelier to respond to political surveys, controlling for their demographic characteristics. Are the “MSNBC moms” now excited to take a poll while they put Rachel Maddow on mute in the background? Like most of the other theories presented here, there’s no hard evidence for it — but it does fit with some well-established facts about propensity to respond to surveys.

Image
Waiting to vote on Election Day in Great Falls, Mont. One possible explanation for 2020’s polling misfire involves turnout — specifically, more of it from Republicans than the polls predicted.Credit...Janie Osborne for The New York Times

The turnout hurt the polls. Political pollsters have often assumed that higher turnout makes polling easier, since it means that there’s less uncertainty about the composition of the electorate. Maybe that’s not how it worked out.

Heading into the election, many surveys showed something unusual: Democrats faring better among likely voters than among registered voters. Usually, Republicans hold the turnout edge.

Take Pennsylvania. The final CNN/SSRS poll of the state showed Mr. Biden up by 10 points among likely voters, but by just five among registered voters. Monmouth showed Mr. Biden up by seven among likely voters in a “high-turnout” scenario (which it ended up being), but by five points among registered voters. Marist? It had a lead of six points among likely voters and five points among registered voters. The ABC/Washington Post showed a seven-point lead for Mr. Biden among likely voters and a four-point lead among registered voters.

It’s still too soon to say whether Republican turnout beat Democratic turnout, but it sure seems possible. In Florida, the one state where we do have hard turnout data, registered Republicans outnumbered registered Democrats by about two percentage points among those who actually voted, even though Democrats outnumber Republicans among registered voters by about 1.5 points in the state. Here, there is no doubt that the turnout was better for the president than the polls suggested, whether they’re private polls or the final Times/Siena poll — which showed registered Republicans with an edge of 0.7 points.

If Mr. Trump fared better among likely voters than among registered voters in Pennsylvania, a fundamental misfire on the estimate of turnout could very quickly explain some of the miss.

Unlike the other theories presented here, this one can be proved false or true. States will eventually update their voter registration files with a record of whether voters turned out in the election. We’ll be able to see the exact composition of the electorate by party registration, and we’ll also be able to see which of our respondents voted. Perhaps Mr. Trump’s supporters were likelier to follow through. We might start to get data from North Carolina and Georgia in the next few weeks. Other states might take longer.

The pandemic hurt the polls. Remember those Times/Siena polls from October 2019 that showed Mr. Biden narrowly leading Mr. Trump? They turned out to be very close to the actual result, at least outside of Florida. They were certainly closer than the Times/Siena polls conducted since.

It wasn’t just the Times/Siena polls that were closer to the mark further ahead of the election. Results from pollsters in February and March look just about dead-on in retrospect, with Mr. Biden leading by about six points among registered voters nationwide, with a very narrow lead in the “blue wall” states, including a tied race in Wisconsin.

One possibility is that the polls were just as poor in October 2019 as in October 2020. If so, Mr. Trump actually held a clear lead during the winter. Maybe. Another possibility is that the polls got worse over the last year. And something really big did happen in American life over that time: the coronavirus pandemic.

“The basic story is that after lockdown, Democrats just started taking surveys, because they were locked at home and didn’t have anything else to do,” said David Shor, a Democratic pollster who worked for the Obama campaign in 2012. “Nearly all of the national polling error can be explained by the post-Covid jump in response rates among Dems,” he said.

Circumstantial evidence is consistent with that theory. We know that the virus had an effect on the polls: Pollsters giddily reported an increase in response rates. High-powered studies showed Mr. Biden gaining in coronavirus hot spots, seeming to confirm the assumption that the pandemic was hurting the president.

But if Mr. Shor is right, the studies weren’t showing a shift in the attitudes of voters in hot spots; rather, it was a shift in the tendency for supporters of Mr. Biden to respond to surveys.

Adding to the intrigue: There is no evidence that the president fared worse in coronavirus hot spots, contrary to the expectations of pundits or studies. Instead, Mr. Trump fared slightly better in places with high coronavirus cases than in places with lower coronavirus cases, controlling for demographics, based on the preliminary results by county so far. This is most obviously true in Wisconsin, one of the nation’s current hot spots and the battleground state where the polls underestimated Mr. Trump the most. The final polls in Wisconsin — including the final Times/Siena poll — showed Mr. Biden gaining in the state, even as polls elsewhere showed Mr. Trump making gains.

Don’t forget the Hispanic vote. There’s one state in particular where the polls were much worse in 2020 than in 2016: Florida, where Mr. Trump made huge gains among Hispanic voters.

What happened in Miami-Dade County was stunning. Mr. Biden won by just seven points in a county where Mrs. Clinton won by 29 points. No pollster saw the extent of it coming, not even those conducting polls of Miami-Dade County or its competitive congressional districts.

Most polls probably weren’t even in the ballpark. The final Times/Siena poll of Florida showed Mr. Biden with a 55-33 lead among Hispanic voters. In the final account, Mr. Biden may barely win the Hispanic vote in the state.

What happened in Miami-Dade was not just about Cuban-Americans. Although Democrats flipped a Senate seat and are leading the presidential race in Arizona, Mr. Trump made huge gains in many Hispanic communities across the country, from the agricultural Imperial Valley and the border towns along the Rio Grande to more urban Houston or Philadelphia.

Many national surveys don’t release results for Hispanic voters because any given survey usually has only a small sample of the group. It will be some time until the major pollsters post their results to the Roper Center, a repository of detailed polling data. Then we’ll be able to dig in and see exactly what the national polls showed among this group.

But if the Florida polls are any indication, it’s at least possible that national surveys missed Mr. Trump’s strength among Hispanic voters. It seems entirely possible that the polls could have missed by 10 points among the group. If true, it would account for a modest but significant part — maybe one-fourth — of the national polling error.

These are the initial guesses. Other theories will emerge. In time, to the extent they can be, all of them will be put to the test. And then we’ll know more than we do now, and can revisit this question.

Nate Cohn is a domestic correspondent for The Upshot. He covers elections, polling and demographics. Before joining The Times in 2013, he worked as a staff writer for The New Republic. More about Nate Cohn

A version of this article appears in print on  , Section A, Page 14 of the New York edition with the headline: What Went Wrong With Polling? Some Early Theories. Order Reprints | Today’s Paper | Subscribe

Advertisement

SKIP ADVERTISEMENT