Polling Data vs Reality
Hugh Hewitt interviewed Peter Brown, assistant director of the Quinnipiac Polls, about some of their recent presidential election poll data. It’s a short interview, but Hugh makes the point that, if the Democrat-to-Republican ratio isn’t close enough to what you’d expect in the upcoming election, then the information is suspect. An excerpt:
HH: Now what I don’t understand this, so educate me on it, if Democrats only had a three point advantage in Florida in the final turnout measurement in 2008, but in your poll they have a nine point turnout advantage, why is that not a source of skepticism for people?
PB: Well, I mean, clearly there will be some people who are skeptics. This is how we’ve always done our polls. Our record is very good in terms of accuracy. Again, remember, we’re asking people what they consider themselves at the time we call them.
HH: But I don’t know how that goes to the issue, Peter, so help me. I’m not being argumentative, I really want to know. Why would guys run a poll with nine percent more Democrats than Republicans when that percentage advantage, I mean, if you’re trying to tell people how the state is going to go, I don’t think this is particularly helpful, because you’ve oversampled Democrats, right?
PB: But we didn’t set out to oversample Democrats. We did our normal, random digit dial way of calling people. And there were, these are likely voters. They had to pass a screen. Because it’s a presidential year, it’s not a particularly heavy screen.
HH: And so if, in fact, you had gotten a hundred Democrats out of a hundred respondents that answered, would you think that poll was reliable?
PB: Probably not at 100 out of 100.
HH: Okay, so if it was 75 out of 100…
PB: Well, I mean…
HH: I mean, when does it become unreliable? You know you’ve just put your foot on the slope, so I’m going to push you down it. When does it become unreliable?
PB: Like the Supreme Court and pornography, you know it when you see it.
"You know it when you see it?" This from a guy who makes his living by hyper-analyzing numbers? Yes, a lot will depend on the actual ratio that turn up on election day, but statistics are adjusted all the time to account for other factors and make all things equal (or as equal as they can be). Why not this factor?
Maybe someone can educate me on this, but just the use of the phrase "you know it when you see it" from a statistician really makes me question the confidence I have in his numbers.
Like this post? Subscribe to my RSS feed and get loads more!