Question Wording is Just the Tip of the Iceberg...
Although David raises a great point, the issue for me with all polls is the provision of sufficient information for the consumer to make his or her own judgment. Whether the question is about the election in November or asked as if the election were held today, the percent of respondents who respond with a "don't know" or "undecided" response are rarely reported by those who conduct the polls. Almost all pre-election polls follow up with questions of leaning; knowledge of how individuals respond to the initial question and to the follow-up question is rarely revealed. Providing estimates from the initial as well as the follow-up question would be revealing about individual preferences, the strength of one's preference, and the state of the election.
In fact, the CBS News Poll conducted prior to the New Hampshire primary did provide information about the fluidity of the race: their press release indicated that 28 percent of Democratic voters' minds could still change. But this piece of information did not make the headlines until after the New Hampshire primary election.
David's point focuses on a specific issue concerning question wording -but I think the issue is larger. Disclosure involves not only provision of question wording, but details about the population interviewed (how sampled, response rates) and in the case of pre-election polls, information about the screening for likely voters. All pollsters are interested in producing an estimate that most closely resembles the actual vote on the day of the election. Therefore, their interest lies in those most likely to vote--including the preferences of those who don't eventually vote will only lead to an inaccurate estimate.
What do we know about these likely voter screens or models? Polling organizations vary in the amount of information provided concerning likely voter screening and modeling. In a year during which we have witnessed record turnout for primaries, have the current estimates of candidate preferences taken into account varying levels of turnout? . To what extent are the estimates we are viewing in October, 2008 a result of old or new voter turnout models?
Knowing the details of the polling methodology may not make headlines, but it would provide for more careful consideration of how various factors impact estimates for the candidates and also aid in understanding why on any given day, you may find discrepant estimates among polls.
I can see why campaigns might want to know how voters are reacting to their candidates (they use internal polling, don't they?) and of course, special interests want to know whether their contributions are going to be economically effective (who wastes money on a loser?), but --
Why should the rest of us care whether polling results are accurate or inaccurate?
October 21, 2008 10:38 AM | Reply | Permalink
Why should you care about polling accuracy? Because pre-election polling is but a small part of the survey and polling industry. Survey data are the foundation for much of the information that shapes public policy, for example, the number of Americans without health insurance or the percent of the labor force without a job. What we know about ourselves--from the television shows we watch to the concerns we have about the future of our nation--are based on surveys.
So even if you don't care about the accuracy of pre-election polls, since decisions that impact your life every day are based on survey data, you do want the methodology to be accurate.
October 21, 2008 11:20 AM | Reply | Permalink
"Do you have health insurance?"
"Do you have a job?"
I think I'm almost ready to be hired by a top pollster.
N.B. Of course we might have to go deep: "Sir, can you tell me the name of your insurer and the title of the plan?" Now, I'm ready!
October 21, 2008 2:24 PM | Reply | Permalink
Polls, of little interest? Perhaps they should be, but:
As any sports fan can tell you, there are people out there, and more than a few, who are simply attracted to a winner. And, whether they know it or not, they make their choices on the basis of who appears most successful. In appealing to this type of voter, being ahead in the polls, or being declared to be ahead by the press, is quite the big deal.
October 21, 2008 2:36 PM | Reply | Permalink
I'm on the list of a researcher in this area and every now and then I'm called to test a product for which I'm paid $75.00 cash. I'm there usually for about 90 minutes, maybe less. The last time I tested there were aobut 30 in our group and we tested a few versions of a lite cracker. We were asked to grade the different versions in different ways.
This was a poll to determine whether or not the public would get this cracker and if they did, in what form.
October 21, 2008 3:59 PM | Reply | Permalink
Aren't you referring to your "focus group" experience?
That's not really a poll, is it?
October 22, 2008 1:32 AM | Reply | Permalink
I have occasionally been polled, and whenever it was something I knew a fair bit about, I was always appalled at the questions and possible answers.
One can often deduce the reason for the poll. Some years ago, I got a call from a pollster asking various questions about local hospitals. I eventually determined that the point of the poll was to figure out which name to use for the merged hospital: two hospitals had recently been bought by Tenet Health Care, and they were going to shut one down (I knew this already, and put that together with the questions). I pointed out to the poll-taker that, if the Hayward Fault ever let go, we would probably need many more hospital beds than we had at that point, and a merger would turn out to be a bad idea. She had not even considered that possibility, and of course that was not an option for a poll answer....
October 21, 2008 10:37 PM | Reply | Permalink