Probabilistic vs. Non-Probabilistic Modern Survey Sampling

Remember that huge panel survey for the November elections out from CBS News and the New York Times? I posted on it here, "CBS News/New York Times Battleground Tracker: Republicans Positioned to Take Senate in November."

The survey uses a non-probabilistic sample, a panel survey to be updated monthly before the elections. Well, it turns out the American Association of Public Opinion Research blasted the format in a statement, and a couple of the political scientists at the Monkey Cage have responded.

See, "Modern polling needs innovation, not traditionalism":

In practice, the probability pollster needs to make massive and changing assumptions about the method of reaching people, as the breakdown of landline-only, cellphone/landline and cellphone-only households switches. There is no known ground truth to how people can be reached and the quantity of people at each phone. As the response rates fall below 10 percent, polls need to make more decisions about how to adjust for systematic differences between respondents and the general population. And pollsters continue to make decisions about their models for the likelihood that a respondent will turn out to vote.

In short, probability pollsters need to make many assumption selections into their polls, just as YouGov does! An important difference is that while YouGov examines their selection issues aggressively and publicly, probability pollsters sometimes ignore the growing lists of selection issues they face. While academics and practitioners alike have studied the issue, traditional probability polling still reports a margin of error that is based on the assumption of 100 percent response rates for a random and representative sample of the population. AAPOR writes of non-probability polling: “In general, these methods have little grounding in theory and the results can vary widely based on the particular method used.” In fact, the theory used by YouGov and in other non-probability polling contexts is well-founded and publicly disclosed, based on the general principles of adjusting for known differences between sample and population.

Yet, oddly, AAPOR’s letter barely mentions methodology but instead focuses on transparency; they accuse the New York Times of obfuscating the methodology. That is odd because Doug Rivers of YouGov is a prolific writer who has detailed the methodology at length and subjected the methodology and results to public transparency that rivals the best practices of major polling companies. YouGov polls fare well when scrutinized along with the major traditional probability polling companies. Doug’s (and other’s) academic papers are published in the top peer review journals. If anything, people on the cutting edge of research are not hiding anything; on the contrary, we are fighting hard to overcome entrenched methods by being even more diligent and transparent...

More.