How Media Polls "Manufacture" Public Opinion

While our focus on this special Talking Points Memo segment has naturally been on election polls, another area of concern is public policy polls. Typically, media polls are designed not to report public opinion but to manufacture a "public opinion" that is interesting and plausible for the news stories - often at the expense of anything that resembles what Americans are really thinking.
The major tactics media pollsters use are 1) asking forced choice questions, to get answers even from people who have no opinion; 2) providing information to respondents in the interview, in case they don't know enough already to offer an opinion - and in so doing taint the sample, so that it no longer represents the general public that has not been given that same information; and 3) failing to provide meaningful intensity measures that could help distinguish firmly held views from the "top-of-mind" views that respondents express in the press of a quick telephone interview.
Typically, the news media do not want to report polls that show a significant segment of the public unengaged and without an opinion on an issue. So pollsters "force" (pressure) respondents to come up with an answer, no matter how lightly held. And if respondents don't know enough, pollsters provide them some limited information.
The net result is that often media poll results conflict greatly with each other, because different polls give different information to respondents, and thus influence them to give different answers. That was the case recently with polls that asked people their reactions to the efforts by Congress and the President to address the economic crisis.
On Sept. 23, a Los Angeles Times/Bloomberg poll reported that Americans opposed the congressional bailout plan by a 24-point margin, 55 percent to 31 percent. That story was cited frequently on various television and radio shows as evidence that Americans were "angry" about the bailout. However, the very same day, a Pew Research poll, conducted over the same time period as the Times/Bloomberg poll, found that Americans supported the Wall Street bailout by a 27-point margin, 57 percent to 30 percent.
In both cases, the polls provided biased information to the respondents, by characterizing the government plan and the situation. The "truth" is almost certainly that a large segment of the population was conflicted over the efforts and could not say for sure if they favored or opposed the plan. But the pollsters did not measure uncertainty, and instead asked biased questions that prompted respondents to give an answer.
A third poll by Washington Post/ABC News, also released on Sept. 23 and conducted over the same time period as the other two polls, found an evenly divided public, 44 percent in favor to 42 percent opposed.
Thus, three different polls on the same subject, all conducted at the same time, with three very different reports.
This is not an unusual situation. Recently, different polls reported contradictory "public opinions" on whether people favored off-shore oil drilling, though oddly very few people were reported without an opinion. In The Opinion Makers, I cite several examples in 2007, showing how public policy polls came to quite contradictory conclusions about the public's support for S-CHIP (State Children's Health Insurance Program), for cutting off funding for the war in Iraq, and for Congress to pass a resolution opposing President Bush's troop surge. In those cases, different polls provided different information to the respondents and used forced-choice questions to extract an answer. It's not surprising the polls contradicted each other.
The polls can do better than this. But pollsters and the media have to be willing to 1) ask neutral questions, 2) provide no information about the issue, 3) allow respondents an explicit "don't know" response, and 4) follow-up with a meaningful intensity question. This process will reveal that on most issues, a substantial segment of Americans, probably from 40 percent to 60 percent, are unengaged and have no firmly held opinion. That, unfortunately, is the truth about American democracy, which media polls routinely hide.
Although there is not room here to elaborate at this time, in my book I suggest that this routine distortion of American public opinion undermines the democratic process.















You are absolutely on point. I was "polled" yesterday and the fellow wanted to argue with me. He sounded like he wanted certain answers that were really talking points.
It left me wondering about all these Newspaper and network polls.
I am worried that we don't know what will happen on November 4th and may be horribly surprised, to the dismay and violent consternation of a lot of people.
October 26, 2008 7:58 PM | Reply | Permalink
I think that polls do a great disservice to our political progress. David Moore provides many of the reasons. I would go further and call into question any and all polls on an issue like the financial crisis.
In fact, that whole "crisis" gives a perfect example of the problem. What happened was not a surprise to at least some experts and the proper short term and long term actions were never debated in any substantive way. Differences in opinion do exist and the discussion is further complicated by key actors holding ideological positions that were shown to be seriously flawed by the event. "I was surprised" arguments do not hold water. There were economists who "knew" but had no real way to become part of the discussion before there was a crisis.
The same type of approach was used in the run up to the Iraq War. You don't have to take any position on whether the right or wrong decision was made. What is clear is that public opinion was being created by those in power and instead of really doing journalism, media took a superficial and inexpensive approach.
We should start calling this propaganda and propaganda metrics. Polling should have a very limited role in any serious discussion. At best it is a crude empirical tool and a really good survey of a complex subject would be very expensive and time consuming.
We are being treated like lab rats or mushrooms; take your pick. I do not know what the solution is but part of the solution is for those who use and interpret polls to take a long hard look at what they are doing. Our educational programs need to be much more candid about the value of this methodology. That should begin in high school. (I know a high school math teacher who does talk about this with his class.)
Having said that. I appreciate this piece. We need more like it and we do not need more pieces about statistical concepts that, in effect, attempt to make some "fact" appear to be "scientific."
October 26, 2008 10:30 PM | Reply | Permalink
The polls can do better than this. David Moore
No, they can't.
Public opinion on any issue is either 1) equally split and therefore, of little interest or 2) strongly supportive of one side of an issue and therefore, subject to change with the next news cycle ("a mile wide and an inch deep").
Trust the pols (they're the pros and their jobs are at risk -- the prospect of defeat goes a long way toward concentrating the mind) and not the polls.
October 27, 2008 11:40 AM | Reply | Permalink
I agree with the author that this is a huge problem that needs to and can be fixed.
What I don't understand is the difference between two of the major tactics listed in the second paragraph. It's clear that using forced-choice formats (yes/no) cause problems (Tactic 1). As I understand it, Tactic 3 (not using intensity measures) seems like pretty much the same thing. Academic researchers generally measure intensity by using a multiple-point scale with a neutral midpoint (e.g., strongly disagree, disagree, slightly disadgree, neither agree nor disagree, slightly agree, agree, strongly agree). It's easy to see how this seven-point scale will provide much more information than a forced-choice (disagree/agree) approach. The use of such a scale is quite easy, and it would solve the problems arising from Tactics 1 and 3.
In any event, we need to be much, much more "literate" regarding the various ways of manipulating information, and Mr. Moore has made a real contribution. He is exactly right in saying that the polls can do better -- they can do much, much better. Knowing what people really believe is extremely valuable information.
October 27, 2008 5:35 PM | Reply | Permalink