DON'T QUIT YOUR DAY JOB

There is a divide among political scientists between those who treat polling or survey data as sacrosanct ("Of course it's reliable, look at how scientifically we collected it!") and those who consider it slightly more accurate than flipping a coin. I fall somewhere in the middle. Polling is riddled with issues that aren't easy to explain away or "correct" with post-measurement methodological voodoo (social desirability and question-order effects, for example) but a dozen polls all pointing in the same direction are a reliable indicator of a trend. I suppose I could describe myself as a believer in Zaller's "Miracle of Aggregation" theory with respect to polling – any one is of limited value, but in quantity they paint a useful picture.

My attititude suggests, therefore, that I believe Obama is going to win. Why? Because it is essentially impossible to find a poll that says otherwise right now. He has won every nationwide poll since Palin opened her mouth and he is the clear trend leader in every important battleground state. His electoral vote total will range between 313 and 375 – a crushing victory – based on aggregated single-state polls. Polling has him ahead in utterly improbable places like North Dakota and North Carolina.

In short, and I say this with due respect to my pollster colleagues, if Obama loses this election the entirety of the contemporary polling industry should be ridiculed into oblivion. Now that literally every single poll is pointing squarely at a solid Obama victory, his defeat would not mean simply that the polls "got it wrong." It would mean that they got it so utterly, overwhelmingly, and inexcusably wrong that the entire art, science, and industry of measuring public opinion will have to be blown up and rebuilt from scratch. This would not be "getting it wrong" like some journalist who picked the Red Sox over the Rays. This would be Dewey Defeats Truman wrong. Maginot Line wrong. They'll Hail Us As Their Liberators wrong. Coke II wrong. Historically, epically wrong.

Could they really be that far off? Well, there are two ways to be wrong in this game – missing high and missing low. Here are a pair of logical, ostensibly plausible scenarios that illustrate how.

  • Scenario 1: McCain Wins – Let's say that there is some characteristic about likely McCain voters that makes them unwilling to admit their support. Maybe they're embarrassed or maybe they just like fucking with the librul media and its polls. Whatever the reason, they're saying "Undecided" when their preference is McCain. So in every state where the polls split along the lines of Obama 47, McCain 45, McCain will come out on top because the 8% of respondents indicating "Undecided" or "Don't Know" are really his supporters.
  • Scenario 2: Obama Hits 400 EV – Polls are often accused of undercounting young, black, and low-income voters (more on that later this week). They also under-represent cell phone users in most cases, although good organizations are correcting for that in their samples now. But for the sake of this argument, suppose that turnout among (overwhelmingly Democratic) college-aged and black voters positively dwarfs anything we've seen before. Both demographics turn out in droves, far in excess of the rate at which they are sampled in polls. Obama not only wins everything he is currently predicted to win but pulls a few "holy shit!"-style upsets in places like Tennessee, Louisiana, and Georgia.

    Is either scenario likely? We can only speculate at this point. I know enough about the guts of big polling operations – and some of the folks involved – to be certain that they have thought of these issues. Gallup et al employ high-level statistical wizards and experts in polling methodology to correct for or avoid such landmines. I have confidence in my colleagues. What I don't have confidence in is the efficacy of quantitative ways to "correct" the inherent limitations of survey-based research. When shove comes back to push, we are still basing conclusions about an electorate of over 180 million eligible voters on the responses of ~800 yahoos who are lonely enough to sit on the phone talking to a pollster (or worse, a robo-dialer) for 15 minutes.

    The error and obstacles inherent in this process means that we shouldn't be shocked if polls are wrong – we should be amazed that they're ever right. But this year, with every single indicator pointing in the same direction, there will be consequences for being wrong. The entire industry can't just chuckle and say "Well, nature of the beast!" Heads will roll, souls will be searched, and we will have to go back to the drawing board. The Smooth Jimmy Apollo excuse from The Simpsons ("When you're right 52% of the time, you're wrong 48% of the time!" "OK Jimmy, you're off the hook.") isn't going to cut it. It's not possible to blow something this badly and simply go back to business as usual.

  • 6 thoughts on “DON'T QUIT YOUR DAY JOB”

    • "It’s not possible to blow something this badly and simply go back to business as usual."

      hopefully the polling is right, and this will be the message voters send the republicans next month.

    • Seems like the meta-problem is that, if the polling is wrong, one can only speculate as to why.

      If, for example, we knew for certain that scenario 1 played out because of some "Bradley effect", then the polling agencies could either adjust for it, or just say "this is a race for which polling is ineffective" sometimes.

      But aren't there cases where someone consistently gets it wrong, but is still chosen for some attribute other than correctness (entertainment value, security, comfort, …)? I mean, norcalfella (on my site) outpredicts both the Vegas line and ESPN et. al. on NFL games. [Yes, of course, he should make money betting it.] But there's a reason the ESPN fools have a paying gig, whereas norcalfella doesn't. (Not sure what that reason is, but now I'm finally getting to my point—it's possible that the objective of a pollster isn't "predicting the answer accurately".)

    • New Hampshire primary, anyone? Maybe Obama just "polls well".

      Also, it appears McCain has regained a lead in both Florida and Ohio. This might just get interesting.

    • The NH result was within the margin of error on a terrible poll (Democrats only in a state with open primaries). I wouldn't call Vegas and place any bets based on that precedent.

      I think it's a slight overstatement to say that McCain has regained the lead. Not coincidentally, the narrowing of Obama's trends in the states you mention is the result of some recent and suspiciously "close" FOX/Rasmussen polls. Hmm.

    • Scenario #1 is entirely possible if the electronic voting machines make it so. What I find encouraging is that most of the polls that usually slant right are showing Obama ahead in so many states.

    Comments are closed.