STUPID QUESTION, STUPID ANSWER

Posted in Rants on February 20th, 2012 by Ed

Polling has gotten much more accurate in recent years, as the field that used to be an art is now a well understood science. By that I mean that we now have a good understanding of response effects, framing, and how to avoid poorly worded or leading questions. I'm hardly an authority on the subject, but I know enough to be staggered by just how terrible some survey questions from major polling outfits can be. This is compounded by the frustration of watching the media present endless public opinion data without the slightest understanding of what the numbers mean or how the questions can influence the results.

Consider the following question from a Feb. 10-13 CNN/Opinion Research poll (n=1,026 adults nationwide ± 3, 228 Catholics ± 6.5)

As you may know, the Obama administration has announced a new policy concerning health insurance plans provided by employers, including religious organizations, and how they handle birth control and contraceptive services for women. Based on what you have read or heard, do you approve or disapprove of this policy?

Compare this to two other pieces of information from the same poll. First, 81% of all respondents and 77% (!!!) of the Catholics disagree with the statement, "Using artificial means of birth control is wrong." Furthermore, 88% (!!!!!!) of Catholics chose the latter option when asked, "Do you think Catholics should always obey official Church teachings on such moral issues as birth control and abortion, or do you think it is possible for Catholics to make up their own minds on these issues?" In light of this widespread support for contraceptive use, the results from the first question – 44% approve, 50% disapprove – appear way too low. It creates the impression that the White House's new policy is quite unpopular.

Compare this to two similar questions from other polls.

CBS/NYT (Feb. 8-13, 2012. N=1,197 adults nationwide. ± 3) asked, "Do you support or oppose a recent federal requirement that private health insurance plans cover the full cost of birth control for their female patients?" A substantial majority indicated support (66% support, 26% oppose). Fox News/Anderson-Shaw (Feb. 6-9, 2012. N=1,110 RV nationwide. ± 3) asked, "The new Obama health care law requires that employer health plans provide birth control coverage as part of preventive services for women. Overall, do you approve or disapprove of requiring employer health plans to cover birth control for women?" They found 61% approval (80% of Democrats, and even 39% of Republicans).

So why did the CNN poll find so little support? Compare the Fox, CBS/NYT, and CNN questions. The other two questions explain what the new policy is, whereas the CNN question simply asks respondents for an opinion on "a new plan" "concerning health insurance" and "how they handle birth control." It does not describe the new policy except to say that it exists and has something to do with contraceptives.

Americans know almost no policy specifics, so asking for an opinion "based on what you have heard" makes no sense. Most respondents will simply offer a response based on whatever information they can glean from the question…in this case, that is likely to be CNN's description of the "Obama administration" policy. In essence, most respondents will read that question simply as, "What do you think about Barack Obama? Yea or Nay?" Unsurprisingly, CNN gets a result (~45% support) that looks suspiciously similar to the President's current approval rating. Fox and CBS, on the other hand, show support that more closely reflects the general public attitude toward contraceptive use.

A lot of right wing blogs and pundits have seized upon those CNN numbers to imply a lack of public support for the new policies, but the results are based on a flawed question. Imagine if they asked "Do you support or oppose the way the Federal government taxes the sale of exotic pets?" Since almost no one will have the slightest idea how or to what extent the government taxes exotic pets, a meaningful answer to this question cannot be given. Respondents will simply pick one of the recognizable parts of the question – Do I like taxes? Do I think it's a good idea for people to have exotic pets? – and respond based on their attitudes on that topic.

Either the folks at CNN and Opinion Research are wildly optimistic about the level of political knowledge and attentiveness to the news of the American public or they lack a basic understanding of how to ask a basic policy question. Regardless, these flawed results are now available to anyone who wishes to distort this issue or to suggest that the public does not widely support the use and availability of contraceptives.