A Problem with Public Opinion Surveys

The notion that a multiple-choice opinion survey will render valid results is highly questionable.

Let’s imagine that we were to survey people on their opinion of the asteroid XR69P in the Migicue galaxy:

How do you feel about asteroid XR69P?

a. I like it.

b. It’s wonderful.

c. It’s too big.

d. It’s too small.

e. It’s too gaseous.

f. It’s too solid.

g. It sucks.

h. other.

Obviously, nearly everyone would choose reponse h. In the case of an obscure asteroid, as with other other utterly unfamiliar material, survey respondents are aware of their ignorance, their indifference and their absolute lack of basis upon which to form an opinion.

On the other hand, there are surveys on topics with which the respondents have some familiarity: 

How do you feel about David Letterman?

a. I like him.

b. He’s wonderful.

c. He’s too irreverent.

d. He’s too commercial.

e. He’s too old.

f. He’s too liberal.

g. He sucks.

h. other.

In this case, 98% of Americans know who David Letterman is. As a result, over 90% of those surveyed will express an opinion including the 65% of them who haven’t watched his show in twenty years and have not had a thought, much less a conversation about David Letterman. These respondents are mostly ignorant, mostly indifferent but forming a response because they are being asked for a response. 

Indifference to a question is not measured by the response that question illicits when posed. To put it another way:

Which is more reflective of public opinion on David Letterman?

a. the fact that  a great majority of the respondents live their lives without ever having a thought, much less an opinion, about David Letterman

or

b. the fact that a great majority of the respondents expressed an opinion about him when asked

Let's say that 80% of the respondents gave positive answers. And let's say that 50% of those never give David Letterman a thought and are only answering because they are being asked. (They are indifferent but, when asked, have more positive than negative feelings about Letterman.)

The pollster would be likely to conclude that the public feels very positively about Letterman. It wouldn't be an exaggeration to say that this conclusion is false. For it counts indifference as a positive reaction.


Now, it makes little difference whether polls of opinion accurately assess attitudes toward a TV personality, but it makes a big difference when the opinions expressed concern public policy and political candidates. For when large swaths of uninformed, uneducated, a-political citizens vote (elections being a form of opinion survey), as has occurred in the most recent case with The Tea Party, the course of a country swings in their direction. It is as if we checked "It's too gaseous." and science, taking our opinions into account, therefore decided to turn its attention to studying (non-existent) asteroid gas.