A Pox on "The Poll of Polls"

Enough already with these "poll of polls". CNN does it, RealClearPolitics does it, and so does Pollster. And so too will a lot of other imitators do the same thing in the next election cycle, if we just let it all slide by right now. Not on my watch. This morning, to take a conspicuous example on my home front, brought news from Pollster.com that Ohio was now back in the tossup column, based on the averaging of the latest polls, apples and oranges alike.
FOX/Rasmussen's robotic, digitally-recorded poll had Ohio as a dead heat with both McCain and Obama at 49% each. The reputable Quinnipiac Poll showed Obama up 50% to 43%. Reuters/Zogby's numbers from their online, interactive poll looked "too close to call", with Obama leading by just two points (49-47). SurveyUSA's robo-poll also had it too close to call (Obama by 48-46). But the University of Cincinnati's Ohio Poll--one with a great record of accuracy (not just because I work there...see it for yourself) had Obama leading by six points (52-46). And last, but not least, among the apples and oranges in the "poll of polls" this morning were the numbers provided by Strategic Vision--which as Mark rightly noted today, is a Republican affiliated firm-- telling us that McCain was up by 2% (48-46) in a race to close to call.
Now maybe I just don't get it, but I thought the only justification for averaging these various kinds of polls was to presume that all of the differences among them is purely random, arising mostly from sampling error, which all sort of cancels out in the aggregate. But each of these polling organizations use systematically different methods of interviewing, screening for likely voters, weighting for party identification, estimates of turnout, etc.--none of which is random and none of which magically cancels out! Maybe we've been lucky with all this averaging business thus far, and it won't kill us on Election Day. But when it's all over, we should take a close look at how all these individual polls performed in forecasting the outcome of the election, and not just average the differences away in another "poll of polls".
Right now it looks like it won't make much difference anyway in how we crunch the numbers in the "poll of polls". David Moore's spot-on. It's not "bloody likely" that most of the reputable polls are way off and that McCain-Palin will somehow manage to pull it all out in the end. If McCain loses Virginia and Ohio tonight, as seems likely from my reading of the reputable polls, Obama's big blue tsunami will roll across the country, resulting maybe not in a landslide of the proportions we witnessed for the Democrats in 1964 and again in the 1974 post-Watergate election, but in a decisive victory nonetheless. If the "truly undecided" break in Obama's favor, it could well be a blowout this evening with Obama-Biden winning 350 electoral votes or more. Not until after the election when we have all the data in hand will we know how much "early voting" and Obama's "ground game" mattered and what it all meant. Wish the exit pollsters well in getting it right tonight, because they're the ones who really have to contend with the impact of early voting on their projections--not to mention the exit poll nightmares of the 2000 and 2004 presidential elections. Buyer beware!














24 hour a day, 7 day per weeek Cable News shows, and constant reporting of polls 5 months before the election with a tsunami of poll reporting as the election nears is like suffering the Death of a Thousand Cuts.
My escape often is M A S H and Seinfeld reruns, unless I can catch a Marx Brothers or W C Fields movie.
November 4, 2008 3:09 PM | Reply | Permalink
Isn't this the same as metaanalysis -- take multiple studies with inconclusive results that were performed with varying procedures, weigh, average and come to a definite conclusion.
If it is good enough for the medical literature, surely it is good enough for political polling.
November 4, 2008 4:10 PM | Reply | Permalink
Princeton to the rescue!
http://election.princeton.edu/
November 4, 2008 5:22 PM | Reply | Permalink
I thought it was worth noting that, although a couple of the poll tracking/election projection sites came very close on the actual electoral vote tallies, *none* of them predicted Obama taking Indiana.
There's an interesting comparison of predictions vs. results at:
http://www.electionprojection.com/projection_scorecard.shtml
November 5, 2008 5:31 PM | Reply | Permalink