Mixed Poll Averages Risky as Handicapping Tool

Carl Bialik takes a look at the prevelence of "mixed poll averages" this election season, raising a red flag on their validity, at the Wall Street Journal:

In the month leading up to last week's delegate-rich California primary, at least a dozen polling firms canvassed the state, collectively calling tens of thousands of households.

Political junkies tracking television, newspaper and online coverage of the voting also heard the names of two main providers of polling data that didn't place a single call: Real Clear Politics and Pollster.com. Both are mashing up surveys from various sources this election year to produce composite numbers meant to smooth out aberrant results. Their methods are criticized by statisticians, but their numbers are embraced by news organizations eager for a way to make sense of conflicting polls.

Numbers from Real Clear Politics, which has been averaging polls since the 2002 congressional races, are used regularly on Fox News, MSNBC's "Hardball," and the Web sites of CBS News and the Washington Post. Pollster, which started combining polls in 2006 and attempts a more complicated mix than a straight average, is featured on Slate and the political Web site, Talking Points Memo.

Stirring disparate pollsters in one pot has its critics. "That's dangerous." says Michael Traugott, professor at the University of Michigan, and author of a recent guide to election polls. "I don't believe in this technique."

Among the pitfalls: Polls have different sample sizes, yet in the composite, those with more respondents are weighted the same. They are fielded at different times, some before respondents have absorbed the results from other states' primaries. They cover different populations, especially during primaries when turnout is traditionally lower. It's expensive to reach the target number of likely voters, so some pollsters apply looser screens. Also, pollsters apply different weights to adjust for voters they've missed. And wording of questions can differ, which makes it especially tricky to count undecided voters. Even identifying these differences isn't easy, as some of the included polls aren't adequately footnoted.

Read the whole thing.

I've cited RCP's numbers in some of my entries, although I don't like mixed averages so much - especially since any old poll seems to be included in the averages, from respectable organizations or not.

Nevetheless, I've focused on general trends in my analyses, sticking mostly to the Republican side, noting potential survey biases or other discrepancies as warranted.

It's probably more reliable to look at a large number of surveys, comparing findings, predicted margins, sampling methods, etc., and then making rough assessments on likely electoral outcomes.

Mixed averaging won't be going away any time soon, you can bet. Horse-race handicapping is too fun for that.