Unlike Magritte, I'm not lying. It really isn't a polling rant. This is about how polling can be useful and how it can be used by partisan hacks to generate misleading "data" to the liking of a paying client.
Think of a poll question like a piece of surgical equipment – it has to be kept perfectly clean. If it gets infected, the outcome is bound to be infected. To generate a good poll question which will give you an honest, unbiased outcome, the key is to avoid giving any information in the question. I can't stress that enough. Any information available in the question will seriously bias the responses.
Let's say, for example, that you are Ford. You want to know what people think about Ford compared to Toyota. If you really want to know the truth, your question look be something like this:
Q. When I say the name "Ford", what word(s) come to mind given whatever you know about their products?
Open-ended questions can admittedly be a pain, and I understand why pollsters don't like to ask them. So the next best question (flawed, but not terribly so) would be:
Q. Given what you know about their products, when I say "Toyota", do you consider their products more reliable than other manufacturers, less reliable, or about the same?
A terrible poll question would start with a piece of information, which by default becomes the basis for how the respondent is likely to consider his answer:
Q. A recent study by Consumer Reports ranked Ford trucks #1 overall in quality. Do you consider their products to be of higher quality, lower quality, or about the same compared to other manufacturers?
And then we get to the absolute worst possible question, one which violates every single criterion separating a scientific poll from pseudoscientific grabass – the question beginning with an unattributed, unsubstantiated piece of conjecture:
Q. Some people think Ford's products are unreliable. Do you consider their products to be of higher quality, lower quality, or about the same compared to other manufacturers?
If you really want to know what consumers think about Ford, you will ask the first question. On the other hand, if Ford calls up your polling agency with its deep pockets and makes it very clear that they need some data to support their new commercials about quality, a disreputable polling agency knows which question to ask (companies like Gallup have too much brand equity to do this, but they aren't immune to asking bad questions anyway).
These issues are no different in politics than with cars. Hack agencies like Zogby and Opinion Dynamics (which barely existed before Fox started paying them) are adept at telling their conservative customers what they want to hear. Consider this actual question from a Zogby poll in 2007:
304. Some people believe that the Bill Clinton administration was corrupt. Whether or not you believe the Clinton administration was corrupt, how concerned are you that there will be high levels of corruption in the White House if Hillary Clinton is elected President in 2008?
An agency called "RT Strategies" asked one of the great hack questions of all time prior to the 2005 State of the Union Address, when Republicans needed to pay to generate favorable data:
Thinking about the war in Iraq, when Democratic Senators criticize the President's policy on the war in Iraq, do you believe it HELPS the morale of our troops in Iraq or HURTS the morale of our troops in Iraq?
OK, note how many different things are being manipulated. First, the respondent might not have had any idea that Senators are critical of the President. Second, they might not know that the critics are Democrats, which guarantees the response of everyone who dislikes Democrats. Third, the only two options are "helps" and "hurts" – neglecting "it makes no difference" – and obviously there's no way to conceive of it helping. Lo and behold, they got 70% of respondents to say it hurts morale. This is approximately like if I asked "Would you consider Ed Burmila to be an awesome blogger or the worst blogger in the history of blogging?" Obviously it's possible that you think I'm mediocre, or good, or kinda crappy, but since you're unlikely to think I am the worst blogger you have ever seen (a pretty extreme statement of dislike) and the only other choice is "awesome"…..
Polls aren't necessarily bad. But they can be manipulated and the way they are used is incredibly irresponsible. Note CNN's hissy fit about how Biden has damaged Obama ("Post-Biden Poll Shows Dead Heat") when this result is no different than the earlier polls showing Obama ahead – IF and only if the margin of error is taken into account. Which it never is. Obama 46%, McCain 43% is in no way different than McCain 45%, Obama 45% if the MoE is +/- 3%. I know that's hard to wrap the brain around, but those two results are exactly the same. The numbers are different but they tell you the same thing: namely nothing.
