The constant complaints from the right about the inaccuracy of polling reflects a deep-seated distrust of Math and Facts and Science that happens to disappear completely whenever the polls say they're doing well. It has been especially amusing this year to listen to their conspiracy theories, since anyone with even a passing understanding of the process (and challenges) of conducting polling today would realize that if anything they are likely to oversample the most likely Trump supporters.

I've always defended, and will continue to defend, the fundamental reliability of the polling. Is every polling firm totally above board and in compliance with AAPOR standards in every single poll they conduct? Of course not. There is always going to be a "pay the piper" element to polling conducted on the behalf of media outlets, parties, campaigns, and politically active groups. But taken as a whole, polling gets it right. Since 2000, polling has done an almost eerily good job of predicting election outcomes provided – and this is an important caveat – we are willing to accept "too close to call" as a valid result. One of the reasons that aggregate polls have become so popular as a predictive tool on sites like 538, Real Clear Politics, and Pollster.com is simple: it's really quite accurate.

Most polling agencies now incorporate some measure of online-administered responses in their samples, but telephone polling of both landline and cell users remains the backbone of the industry. And polling industry people will tell you that random samples do not materialize from random phone dialing. In theory it should, but here's the thing: while any phone number has an equal chance of being dialed by a random number generator, there is a bias to who actually 1) answers the phone and 2) stays on the line to participate once they realize that this is either telemarketing or a survey. One of the reasons pollsters have such a hard time getting young people, low income people, blacks and Hispanics, and other demographic groups in their samples is that those people are the least likely to stay on the phone and participate even if the survey team manages to reach them. For younger and cell-only users, they're unlikely to even answer when they see an unrecognized number.

So, to produce random samples survey researchers resort to a lot of…statistical adjustments. Weights are given to respondents until the sample roughly approximates their target population. Firms differ in whether their "ideal" sample is the population as a whole or their best guess at the demographics of the electorate that actually shows up to vote (different guesses about which groups of people will turn out and in what numbers are one of the key reasons that polls often vary slightly among firms). They get away with this largely because 1) Young people generally do not vote, so getting a sample with few young voters is survivable, 2) African-Americans and Hispanics are pretty monolithic in their Democratic preferences, so a small number can be weighted to represent a larger population without much error, and 3) old white people are both the most likely to respond when phoned and the most likely to vote.

Why is it that the majority of people responding – not being called, but actually responding – to polling calls are old? Well. They're more likely to be home when a pollster decides to call, especially after retirement. They're more likely to want to vent their opinions at someone. They're less likely to reflexively hang up on an obvious cold call because they developed their phone habits in an earlier era. And, I'm sorry to say, a lot of them are just lonely and want to talk to someone. The challenge of polling in this era certainly is not "How do we get enough likely Trump supporters in our sample, especially white ones over 55?" The challenge is getting anything other than that.

To that end, right wingers who complain about polling are correct: the unweighted sample drawn by polling firms is not truly representative of the demographics of the nation or even of the electorate. It is, if anything, over-representing the people who are most likely to be conservatives today.

25 thoughts on “RANDOM SAMPLE BLUES”

  • Emerson Dameron says:

    Some of the confusion stems from the fact that conservatives have their own polls promising a Donald landslide, which means those pencil-necked liberal pollsters are obviously lying.

    I have a feeling Trump 's base will not take this election particularly well.

  • So polls reflect best guesses, hunches and the unknown in an attempt to reflect reality and sometimes get it right.
    I'm cheered to know that we share a ride on the Titanic.

  • "… a deep-seated distrust of Math and Facts and Science that happens to disappear completely whenever the polls say they're doing well."

    I don't think that's quite right. When the polls look good for them they pat themselves on the back because they're EVEN doing well in those TOTALLY UNFAIR LIBERALLY BIASED polls.

    It's analogous to the way that, when The NYT reprints their spin verbatim, as they do two or three times a week, it's not evidence that the NYT isn't liberally biased, but a sign that "even the totally liberally biased NYT can't ignore this!"

    Lowering the bar for themselves. It's all they do. It's all they've ever done.

  • My main gripe about telephone polling is that they cannot, or are under orders to not, tell you who is behind the poll and when and where I can see the result. Push polls are obvious and are easily ignored. If I could be assured it's an independent poll – some poll that I'm familiar with that wants unbiased results – I'd likely not hang up but all I get is the name of the company that runs the call center making the calls.

  • @mago

    Yes, that's one way of framing it. Of course, every branch of science needs to rely on "best guesses" and "the unknown" to some extent. The fact that we don't know everything perfectly doesn't mean that our best guess isn't likely to be damn close.

    If polls were perfect, there would be no need to hold an election. But the number of elections that have been extensively polled, and where the polls didn't reasonably forecast where the election ended up, is pretty small.

  • c u n d gulag says:

    I worked part-time for a company that did product, and conservative political polling – this was back in '09-'10.
    And Ed is exactly right in what he says.

    Further, I used to do the demographics right up front, instead of at the end, so if I got over 50% on a stupidly long poll done, I could turn it in and get credit for it.

    What was hysterical, and pathetic, if one of the questions were the news sources of the pollee, and it was (DUMB)FUX "news", Drudge, or Reich-Wing talk radio, I didn't actually need to ask the questions, because I'd already know the answers. But, I did the right thing, and kept asking questions.
    Only one guys surprised me – he was for a woman's right to choose!
    I damn near fainted!!!

  • @ waspuppet Says: "Lowering the bar for themselves. It's all they do. It's all they've ever done."

    Exactly; and the rest of us who like to think above the crotch are forced into a perpetual game of political limbo with these low-ball obstructionists.

  • The constant complaints from the right about the inaccuracy of polling reflects a deep-seated distrust of Math and Facts and Science that happens to disappear completely whenever the polls say they're doing well.

    They hate science, yet envy its aura of authority. They prefer not to flatly deny science, but instead manufacture junk science which supports their preferred conclusions. The same pattern appears in creationism and global warming denial.

  • Dave,

    What you mention is unsurprising. I wouldn't want to introduce yet another source of bias by telling my respondents who is conducting the poll. That would make any poll conducted by a major media network meaningless, as people self-select based on their political/media preference.

    Sure, telling people AFTER they complete the survey who's conducting the poll or where they can find the results would not necessarily be a bad idea. But beforehand? That's just making an already difficult job (polling a diverse enough sample) even more difficult.

  • What I hate about answering polls is when the questions are yes/no but I can't honestly answer in such simple terms. The last one I answered, the first question was "Do you think the country is headed in the right or wrong direction?" On what issue? Can I answer NO direction? I decided the question was meaningless and I couldn't proceed further, so I apologized to the poor schmuck calling me and hung up.

  • As an Ur-Millenial, being born in 1982, I absolutely let unknown numbers go to voicemail. If it's actually important, a message will be left. That said, I think out of curiosity, I answered a survey only once this election season.

  • PhoenixRising says:

    Additional factor in whether a call from an unknown number will be answered: Is the owner of that phone in debt to anyone?

    If an unknown incoming call could be from a collection agency or vendor you're juggling payments to, you're not picking that up. Which introduces another skew to the 'random' sample, to the extent that people who are broke or in debt vote at a higher rate than they answer polls (which I'd guess is 0%).

  • @Nate, it's not just the Millenials who don't answer the phone for unknown numbers. With all the scammers and other garbage calls, most people I know don't answer because if it's important, the caller will leave a message.

  • @Emerson Dameron

    Shades of the 'Great Unskewing' that Romney's team engaged in in 2012.I *almost* feel sorry for those putzes over the sock to the gut that reality on election night must have been to them. Almost.

    Sadly Trump's supporters will likely not snap into reality, but retreat into even deeper paranoid fixation. That has the distinct potential of being horrifically dangerous.

  • The Washington Post had an article yesterday on one of Trump's deplorables in Pennsylvania; a woman who worked on the railroad, was harassed by her male co-workers, and blames the government (because sure, that makes sense). She descended into evangelical fundamentalism and mental issues (in and out of mental health programs, gets arrested a lot). She LURVES Donald Trump and is convinced he's going to save 'Murkkkuh, and she's also certain President Obama founded ISIS, is married to a man, and travelled back in time to place phony birth announcements in a Hawaiian newspaper.

  • @Katydid:

    As a resident of Pennsyltucky, let me tell you, these people loathe the government an awful lot for people that are totally dependent on it. I live in Harrisburg, the state capital, which means that a ton of people are employed by the State or Federal governments, and then complain about the taxes THAT PAY THEIR SALARIES.

  • @Ed, Charles Pierce (if you're reading this)

    Charles Pierce, October 3rd, 2016

    The great failure in this year's coverage is the failure to attach Trump to the Republican Party as the logical—nay, inevitable—product of three decades of deliberate political and policy choices. If El Caudillo del Mar-A-Lago augurs in, this is going to enable the Republicans to claim that he was an aberration, and not a creature of their own, decades in the making.

    Khaled Says:
    September 13th, 2016 at 4:08 pm
    Trump is the natural, logic, inevitable Republican candidate. A "politician" who isn't a "politician", a "strong leader" (who's actually not that strong or smart, but is playing the part), and has completely divorced himself from reality. Republicans have been playing fast and loose with facts and reality for at least a generation, if not longer. Thank Saint Reagan for bringing it all to national politics. Ever since the Laffer Curve, the Republican Newspeak of Economics and Political Philosophy has dealt with fiction that panders to the rich and uses dog whistles to connect to the lower and middle class white voters. Cutting taxes means more revenue! Deficits are bad when Democrats are in office! Government spending is bad unless it's on weapons! Government is terrible, so elect me to govern! Patriots are for sedition against democratically elected governments! Racism doesn't exist to keep minorities down, but invisible government regulations keep me and my fellow upper-middle class white people from being rich!

    I know I borrow "Vulgar Talking Yam" from you Pierce, but if you're going to borrow my descriptions of Trump, you can PayPal me or at least give me some credit, fool. I mean, stealing Ed's "Hair Trigger" article is bad enough, but to mine comments?

  • Pierce has been on a roll of late. Yesterday he pointed out the frightening prospect of a smarter, "better" Trump type next time around. And that's assuming the Giant Evil Baby (tm Laurie Penny) doesn't win the election in a couple of weeks. That whole "Clinton's chances of losing are approx. those of an NFL placekicker's missing a 47 yard field goal" meme is pretty much keeping me up at night.

    @katydid, I also read that WaPo article about the crazy lady in PA. Yes, she's nuts, but her real descent seemed to be triggered by having a judge throw out her jury-awarded $450,000 sexual harassment settlement. I think I might lose it too if some gubmint official yanked half a million bucks from me.

  • @Khaled, we drive through Harrisburg on the way to upstate NY to visit my in-laws. We've driven the highway on both sides of the river to get from 83 to 81 to 15 to continue north. Lotta really gorgeous old homes, but also a lotta poverty. I can believe they curse the gummint while cashing their gummit handout and denouncing the blah people for being "takers". Also, does *everyone* in York work in Baltimore? It sure seems like it driving south on 83.

    @Geoff; The person who shared the article with me worked at a psychiatric hospital in the 1970s and commented that she perfectly fits the bill for the type of patient they had in there long-term; violent, delusional, and a danger to herself and others. Trump's people.

  • Emerson Dameron says:


    I really do find it quite sad.

    After Romney/Ryan's "shocking" defeat, certain conservatives seemed to do some relatively sincere soul-searching. Their primaries couldn't be Reality TV free-for-alls. They desperately needed to make their less reptilian ideas palatable to middle-class and minority voters, particularly Hispanics.

    And the result… They nominated a child powered by pure bigotry who has already set the party ablaze and will take the rest of the country with it if he and his stupid and despicable followers don't get all the stuff on their Christmas list.

    I feel bad for conservatives in the same way I feel bad for Kim Kardashian this week. I never much cared for them and still don't, but this is nonetheless horrifying and should never happen to anyone.

  • @Katydid

    Yes, almost everyone in York works in Baltimore. It's cheaper than Baltimore, I know people who built a fancy McMansion near York for the price of a 2 bedroom ranch in Baltimore. Of course, the commuting cost will eat up the difference, but whatever. Harrisburg has some really nice stuff and a few blocks of nice-ish stuff near the Capital, but yeah, there is a ton of poverty in the city itself.I live in the burbs, and the Trump signs are going up in my neighborhood and I keep trying to get my dog to pee on them. It's the fear of the "blah" people that drive the love of Trump around here, and a ton of people who never go into the city who think it's the "war zone". Once you get north of the Harrisburg area on 11/15, you get into Pennsyltucky real, real quick.

  • @Khaled, thanks for the info! I always find it hilarious on the way to upstate NY that once you pass Harrisburg, once you leave the river and head more inland, past the Trading Post, you reach a stretch of maybe 10 miles where all you see are porn shops, porn shops, porn shops, the odd little diner, more porn shops, porn shops, porn shops. What is it with Pennsyltucky and their love of porn?

  • I'm a who-you-callin-old,-grasshopper white person. I always vote. I never answer the phone.

    Even in the demographic they think they're reaching, they're undersampling the lefter wing.

  • I'm not voting for Mr. T, but I notice a historical pattern that the Left has been consistent with in characterizing Rs since 1960

    1960 Nixon – Demon
    1964 Goldwater – Super Demon
    1968 Nixon – Demon
    1972 Nixon – Demon
    1976 Ford – Dufuss
    1980 Reagan – Dufuss
    1984 Reagan – Dufuss
    1988 Bush 41 – Demon
    1992 Bush 41 – Dufuss – No Nude Texans
    1996 Dole – Old Old Dufuss
    2000 Bush 43 – Dufuss
    2004 Bush 43 – War Criminal Demon
    2008 McCain – Semi Demon-Dufuss
    2012 Romney – Dollar Demon
    2016 Trump – Full Demon Dufuss


Comments are closed.