With this election, math once again messed with the magic* in a media stalwart. Television pundits, usually with the authority left over from past political victories, turned out to be inferior seers compared to fast-moving analysts armed with a raft of polling data. The Times' own Nate Silver appears to be the biggest winner of all.
But other math, abetted by technology, could mean trouble down the line for our prognosticating overlords. Traditional polling is getting more expensive and less reliable. The emerging online alternatives are promising, but have problems of their own. Problems with the polls may also mean problems for the people who read them. (Nate Silver made a comparison of polling accuracy last week.)
The fundamental difficulty has to do with changes in phone technology and human habits. Much of the polling data you see comes from phone calls. Caller identification has made it easier to ignore calls from polling outfits. Cell phones have call er i.d., and people are likely to be using them from any number of places, where they don't want to be disturbed.
Last May, the Pew Research Center published a report which said that the number of households responding to phone polls has fallen from 36 percent in 1997 to 9 percent today. If this trend continues, at some point response rates will be too low to show good representation.
Even if pollsters do get through, and convince people to cooperate with an in-depth poll, taking these kind of surveys to an increasingly mobile population is more expensive. A 1996 Federal law states that calls to cell phones have to be hand-dialed, not generated by computer. That increases the time in getting the answers.
A study published last spring looked at an effort by the Department of Housing and Urban Development to survey rents. It found that the cost of obtaining one completed survey ranged from $77.18 for a call to a landline phone to $277.19 for a call to a cell phone.
While it is not clear that this study is a perfect match for the costs a political poll, it is clear that calling the mobile population is expensive. That makes follow-up and in depth polls, which are more valuable, less attractive.
“The ultimate question is, how representative are you of the population?†says Michael McDonald, a professor of statistics at George Mason University who studies polling. “I tend to trust organizations that go the extra mile, with personal interviews, calls, and multiple call backs. Fast polls are a strategy if you want to make news, but they aren't as good.â€
One alternative is to rely more on Internet-based surveys, something the pollsters at Rasmussen Reports and other outfits already do. Prof. McDonald says using Internet data, however, “trades one set of biases for another. We don't have full Internet coverage, and not everyone uses computers.â€
Still, as more people get online, the Internet-based pol ls get much better. SurveyMonkey, which sells tools for all kinds of collective voting, carried out over several months an online presidential poll that had 96 percent accuracy, compared with the actual results of the vote.
“We looked at nine battleground states over 11 weeks,†says Philip Garland, vice president of methodology at SurveyMonkey. “On the day before election day alone, 60,000 people took the survey.â€
The cost per person was negligible, he says, and the results may be more illuminating. “We got twice as many ‘don't knows' compared with phone or personal surveys,†says Mr. Garland. “When people are asked questions by a person, they feel like they should make a choice.†Still, like other pollsters, the online service was surprised at the turnout by Latino and African-American voters, indicating the survey isn't perfectly capturing the national population.
SurveyMonkey, which didn't make money off this poll, plans to continue the work for the 2014 midterm elections, and will make its data available to the public. “We expect to get a lot of interest from political organizations,†says Mr. Garland.
Just in case you thought this election thing was over.
*Note: A saltier version of the phrase “messed with the magic†was supposedly uttered by an old-media bigshot when he first toured Google, and learned how its algorithms could make advertising both cheaper and more efficient.
