Polling has gotten much more accurate in recent years, as the field that used to be an art is now a well understood science. By that I mean that we now have a good understanding of response effects, framing, and how to avoid poorly worded or leading questions. I'm hardly an authority on the subject, but I know enough to be staggered by just how terrible some survey questions from major polling outfits can be. This is compounded by the frustration of watching the media present endless public opinion data without the slightest understanding of what the numbers mean or how the questions can influence the results.
Consider the following question from a Feb. 10-13 CNN/Opinion Research poll (n=1,026 adults nationwide ± 3, 228 Catholics ± 6.5)
As you may know, the Obama administration has announced a new policy concerning health insurance plans provided by employers, including religious organizations, and how they handle birth control and contraceptive services for women. Based on what you have read or heard, do you approve or disapprove of this policy?
Compare this to two other pieces of information from the same poll. First, 81% of all respondents and 77% (!!!) of the Catholics disagree with the statement, "Using artificial means of birth control is wrong." Furthermore, 88% (!!!!!!) of Catholics chose the latter option when asked, "Do you think Catholics should always obey official Church teachings on such moral issues as birth control and abortion, or do you think it is possible for Catholics to make up their own minds on these issues?" In light of this widespread support for contraceptive use, the results from the first question – 44% approve, 50% disapprove – appear way too low. It creates the impression that the White House's new policy is quite unpopular.
Compare this to two similar questions from other polls.
CBS/NYT (Feb. 8-13, 2012. N=1,197 adults nationwide. ± 3) asked, "Do you support or oppose a recent federal requirement that private health insurance plans cover the full cost of birth control for their female patients?" A substantial majority indicated support (66% support, 26% oppose). Fox News/Anderson-Shaw (Feb. 6-9, 2012. N=1,110 RV nationwide. ± 3) asked, "The new Obama health care law requires that employer health plans provide birth control coverage as part of preventive services for women. Overall, do you approve or disapprove of requiring employer health plans to cover birth control for women?
" They found 61% approval (80% of Democrats, and even 39% of Republicans).
So why did the CNN poll find so little support? Compare the Fox, CBS/NYT, and CNN questions. The other two questions explain what the new policy is, whereas the CNN question simply asks respondents for an opinion on "a new plan" "concerning health insurance" and "how they handle birth control." It does not describe the new policy except to say that it exists and has something to do with contraceptives.
Americans know almost no policy specifics, so asking for an opinion "based on what you have heard" makes no sense. Most respondents will simply offer a response based on whatever information they can glean from the question…in this case, that is likely to be CNN's description of the "Obama administration" policy. In essence, most respondents will read that question simply as, "What do you think about Barack Obama?
Yea or Nay?" Unsurprisingly, CNN gets a result (~45% support) that looks suspiciously similar to the President's current approval rating. Fox and CBS, on the other hand, show support that more closely reflects the general public attitude toward contraceptive use.
A lot of right wing blogs and pundits have seized upon those CNN numbers to imply a lack of public support for the new policies, but the results are based on a flawed question. Imagine if they asked "Do you support or oppose the way the Federal government taxes the sale of exotic pets?" Since almost no one will have the slightest idea how or to what extent the government taxes exotic pets, a meaningful answer to this question cannot be given.
Respondents will simply pick one of the recognizable parts of the question – Do I like taxes? Do I think it's a good idea for people to have exotic pets? – and respond based on their attitudes on that topic.
Either the folks at CNN and Opinion Research are wildly optimistic about the level of political knowledge and attentiveness to the news of the American public or they lack a basic understanding of how to ask a basic policy question. Regardless, these flawed results are now available to anyone who wishes to distort this issue or to suggest that the public does not widely support the use and availability of contraceptives.
c u n d gulag says:
I'm sorry, but I don't think polling has gotten more accurate – I think it's LESS accurate now than it was even a decade ago.
From October of 2009, until August of 2010, I worked part-time for a Republican polling company here in my home town.
The bulk of the people who answer their phones to polling companies, and answer any questions, are over 60
c u n d gulag says:
Jeez, the bulk of my word-turds didn't register!
Ed must have a spam filter that knows fatuous gasbags when it reads them.
If you're interested, here's the rest:
– with most of them over 70-75. I think most of them are glad that someone’s calling them, let alone asking their opinions.
The younger the age, the less you were able to get a hold of them. Either they had Caller ID and didn’t pick-up, or cell phones, whose numbers we didn’t have access to. It's to the point when you call and a young person answers the phone and is willing to take the poll, you almost fall out of your chair.
That’s why I take any phone polling numbers with a huge grain of salt, because of this. The statistics are skewed to the older, generally more conservative, part of our population.
And internet polls have the opposite result – they’re skewed to the younger demographic.
When everyone had a land-line, and before Caller ID, I think phone polling was much more accurate. Since those days are long gone, I have less and less faith in the phone results.
Now, it’s almost like 1948 when the polls predicted that Dewey would easily beat Truman. The phone polling used to come to this result was in error. In 1948, not every home/apartment had a telephone, so they were polling people who were in the higher than normal income bracket – and those people skewed Conservative then, as the ones who still have a land-line do now. I don’t know of anyone under 30 who’s still got a land-line. Unless, of course, they’re living with their parents. Hell, I’m 54 in a little over a week, and I’ll probably be ditching my CELL phone, once my final unemployment payment ends in a month. We’ll still have a land-line. Not that employers have exactly been ringing either phone off the hook. I think I have a better chance of a Sports Illustrated super-model calling me for a date, than I have of being offered a job.
Don't get me started on the questions!
The people who write them frequently have an agenda, and frame the question to get the answer they want.
And I could write a 10,000 word rant on the genuine evil of push-polling.
But, I'll spare you.
Major Kong says:
I'm reminded of how Stephen Colbert would ask his guests:
"Was George W. Bush a great President or the greatest President?"
Middle Seaman says:
Polling got much; that's a fact. As in every field, there inapt, biased and evil doers.
Then you have the pretenders.
anotherbozo says:
c u n d validates my suspicions. I myself never take polling calls, and I'm almost 70. Either their demand for commitment exceeds my interest ("this will take about 10 minutes") or a "public opinion survey" winds up being a commercial poll ("Well, if there WERE a Pizza Hut in your area, would you frequent it?") or thinly disguised campaign research ("do you support Mike Bloomberg for a third term? do you like another candidate better?"). Bah, humbug.
The public's temperature is taken so often these days as to lose sight of the fact that more and more effort is expended on less and less intelligence, ie, the ignorant, docile goofballs that comprise the voting public.
Rick Massimo says:
The only political poll I've ever been called for took about 10 minutes to get around to the question "Would your support for (candidate I said I supported) be affected if you knew that he approves of a plan to (something along the lines of killing puppies)?"
I told the guy that I knew exactly what he was doing. He asked me to stick to the question. We went back and forth until finally I said "OK, I'll play along with your dishonest question: YES, I support (Candidate)'s plan to (something along the lines of killing puppies). There ya go."
I think issue polls by news organizations, particularly "news" organizations like Fox and CNN, are doubly worthless: After 23 hours of telling you that the Scary Deficit Monster is the most pressing problem facing America, they ask their viewers what the most pressing problem facing America is. The answer then justifies their spending another 23 hours telling us the Scary Deficit Monster is the most pressing problem.
c u n d gulag says:
Besides, who wants to hear from CNN – CELEBRITY News Network?
I'll never forget the Senate fight over Habeas Corpus under "Baby Doc" Bush's dictatorship.
Was CNN on top of of it?
NO!
Instead, they spent almost a whole week, 24X7, on Anna Nicole Smith's death.
I guess Wolf and "The F*CKING Worst F*CKING Political F*CKING Team In The F*CKING History Of F*CKING Broadcast F*CKING Media," had hard-on's for Anna, and didn't care if her corpus was dead or alive.
Dookie says:
Let's see if I understand the logic here…if I believe or support something but I don't support the federal government passing a law mandating everyone else do as I believe, then that is a contradiction? Really??
Interesting…
Loofah says:
I've often thought that the greatest and simplest act of revolution against today's poll-driven political positioning and its dog-chasing-its-own-tail news coverage would be if everyone, simply and without exception, refused to respond to any polls. No data for the poll-takers, no info for the candidates to shape their positions around, no easy content for the talking heads to fill up their shows with. Maybe then politicians would have to make clear what they actually stood for, cable news would have to actually report some real news, and our political dialog might return—maybe, just maybe—to one based on the discussion and evaluation of actual facts and honestly-held convictions, rather than the regurgitation of uninformed opinions that it has become.
Bernard says:
a landline? what is that! lol. angry old white men are getting their revenge now.
Denn says:
Garbage In, Garbage Out.
Juniper says:
Seriously, I think the only people who use a phone to do anything anymore are people over 60. Most people younger than that will use the internet to pay bills, make purchases, etc.
JK says:
The other possibility is the wording of the CNN pol was not an accident. Creating a poll that shows a level of controversy that does not exist allows CNN to keep flogging this dead horse for ratings. No one is going to talk about the foxnews poll, they are going to cite this piece of shit cnn poll instead. Eventually it may even become controversial if they try hard enough and swing a few viewers over to the other side.
Rick Massimo says:
The sad implication of these polls, and other similar situations, is that you'd like to think the politicians reading these polls would care whether they were accurate.
You'd like to think any office-holder or candidate will turn to an aide and say, "Tommy, dive deep into these numbers, and the way the questions were written. This stuff can be deceptive, and we don't want to be on the wrong side of what the public REALLY believes."
Nope. The Christianists found a poll that says what they want it to say, and they're gonna use it as a billy club from now through November and beyond.
Hawes says:
I wouldn't say the CNN poll is useless at all. I just think it measures – as you point out – that people don't really know what the new policy is. If I'm a health official or a political operative in the Obama administration, I have to look at the fact that people don't really know what the contraception rule is, but when they do, they like it.
I think progressives in general think that once we have explained our positions, they will popular with majorities of the population.
But we never reconcile ourselves to the fact that it's awfully damned hard to explain our positions.
This poll simply clarifies this relationship between good policy and public ignorance.
Fiddlin' Bill says:
Polling has long ago merged with "meta polling," such as push-polling, wherein a seemingly scientific result is actually created to conform with whatever opinion is needed for propaganda purposes. It's not at all clear, that is to say, just who is legit in the polling bidness.
Number Three says:
As someone who runs polls for a living (but is reluctant to describe self as "pollster"), let me assure you that one of the most frustrating things about the business is that every last single person you meet thinks that s/he could do your job. This is one of those areas where "common sense" misleads even smart (in other fields) people.
My guess is that this "question" was generated by the suits at CNN, and that the pollsters were told to ask it, regardless of their professional objections. This happens more often than one might expect. I've mentioned this here before– at some point, telling the client that they are wrong just gets tiresome.
A says:
"Would your support for (candidate I said I supported) be affected if you knew that he approves of a plan to (something along the lines of killing puppies)?"
I have always contended that you can twist any question as evidenced above. & you can twist any of the resulting mathematics. I took a statistics class in college ("Problem Solving for Managers" I was not pleased to realize that it was a statistics class!) & we looked into nursing homes, 6 groups in the class had to make the best case for Govt., Private, or religious care – everyone could make their group sound like the best choice while similalry maligning the others!
& when discussing divorce rates, if you tally Liz Taylor along with my grandparents, more than 50% of the population divorces, when in fact 2 of the couples remained married their whole lives to the same person.
bb in GA says:
Statistical manipulations w/ numbers are just tools to put in service of whatever goals you have. (Check out how the US government came up with the famous 3000 excess deaths per year from second hand cigarette smoke)….
just like I can use an 'old school' framing hammer to raise some walls for a nice house or I can punch quarter sized holes in your skull with it if you can't get away from me or neutralize me.
//bb
Mike says:
@Dookie: if I believe or support something but I don't support the federal government passing a law mandating everyone else do as I believe, then that is a contradiction?
That would be a really good point, if the government were mandating that everyone use contraception or get an amniocentesis. (Yes, that's right, Rick Santorum is also against pregnant women having an amniocentesis.)
James M. Martin says:
This is precisely how the National Socialist party in Germany got its start and spread its message, by lies. Goebbels admitted, "if you tell the lie often enough they will believe it is true," and he hinted that the bigger the lie, the more readily the populace will seize upon it as fact. If you are going to watch an hour of Fox, you need to get an hour of MSNBC as well. Somewhere there is truth. Or something like it.
Ellie says:
"…endless public opinion data without the slightest understanding of what the numbers mean or how the questions can influence the results."
Well, I got called for a big name, mainstream] poll the othe week.
[several questions, then..]
"Do you approve of disapprove of the job Barack Obama is doing?"
"Disapprove"
[blah blah blah, more questions]
"Would you consider yourself very conservative"…etc., through to…"very liberal."
"Very liberal'
I read about same poll a few days later. Do you think for one second that they broke down Obama's disapproval rating between disapproval from the left and disapproval from the right?
OF COURSE they didn't. They never do.
I don't have a point….I'm just irritated.
Ruthie says:
What JK Says at 2/10/12 at 2:06
Plus the pollsters cherry-pick respondents based on zip code demographics. I live in a zip code populated primarily by mostly retired, hideously conservative, and semi-senescent old farts. The CNN and FOX people must have us on speed dial as their token liberals.