This is an archive of the old Stones Cry Out site. For the current site, click here.
« Don't you just love to vote? | Main | Mixing God & Science »
August 23, 2005
Statistical Significance v. Validity
Hugh Hewitt mentioned “statistically significant” twice in as many posts about Patrick Ruffini’s August Straw poll.
The words of Inigo Montoya immediately came to mind: “You keep using that word. I do not think it means, what you think it means.”
I know Patrick Ruffini knows a thing or two about stats, so I had to check it out. Sure enough, there it was…er…was it? Here’s the line: “If this poll is as big as the last one (@13,000 responses), we'll have a statistically valid sample of online activists not just nationally, but in most of the fifty states.”
Notice that Patrick used the word “valid” and not “significant.” Is there a difference between statistically significant and statistically valid? Well, yes. There is a big difference.
Statistical significance is a function of sample size and degree of confidence. 13,000 responses to a simple random sample of population over 100,000 would yield a margin of error of either 0.8% or 1.1% depending on whether one wants to be 95% or 99% confident in the significance of the results. (Click here for a neat sample size/confidence interval calculator.)
Determining the validity of a survey is another thing. Statistical validity can be defined as “the degree to which an observed result, such as a difference between 2 measurements, can be relied upon and not attributed to random error in sampling and measurement.”
Robert Groves’ Total Survey Error framework describes four components to survey error: 1) sampling; 2) nonresponse; 3) coverage; and 4) measurement. Statistical significance addresses only sampling error. Statistical validity incorporates all four of Groves’ TSE components.
First off, Patrick’s survey is not a random sample of likely voters in the 2008 primary and therefore thinking in terms of statistical “significance” is misleading; however, the potential coverage error presents the greatest threat to the poll’s “validity.”
Coverage error is the error introduced to a survey when a segment of the population that the survey intends to represent is not included in the sampling frame. That is, what portion of Republican primary voters are not internet users, or more specifically, readers of the various internet sites that might advertise Ruffini’s straw poll? It doesn’t matter how large the sample size is if the population being sampled isn’t representative of the population the survey is intended to represent. The results may be interesting, but claiming statistical validity is not possible based on sample size alone.
For the record, I participated in the straw poll and selected Mass. Governor Mitt Romney, who isn't doing particularly well at the moment. Don't let my post stop you from playing along. As I said, the results, although perhaps not statistically significant or valid, are interesting; especially if you contend that internet consumers of conservative political news set the agenda for non-internet savvy conservatives.
Reference:
Groves, R. M. 1989. “Survey errors and survey costs.” New York: John Wiley.
Posted by Rick at August 23, 2005 01:45 PM
Trackback Pings
Comments
Thank you for pointing this out. I am so incredibly tired of people across the political spectrum misusing the terms of science, and statistics in particular. It seems they can't even be bothered to learn the basics about which they are speaking, but that would seriously limit their universe of punditry, wouldn't it?
It's particularly disturbing in Hugh's case because he litigates land use and evironmnetal law. Is he really that ignorant of the foundations of many of his arguments?
Posted by: Jon Gallagher at August 23, 2005 02:42 PM