Recently, my friend Daniel Stublen made the argument that reporters should always include the margin of error when citing polls. This led to a good discussion about what the margin of error really is, but I wanted to dive deeper into it, here.
Polls are bound to have sampling error (the error that occurs because you ask a subset of the population, rather than the entire population). The margin of error measures the maximum amount by which the sample results are expected to differ from those of the actual population. The lower the margin, the more confident you can be in your estimation.
Let’s consider a real-world example. Between October 25-30, 2019, the New York Times Upshot/Siena College asked 439 Iowans “Who is your first choice?” and 22% responded with Elizabeth Warren. The margin of error on this was 4.7 percentage points.
So the interpretation of this poll is: “At least 95 out of 100 times, \(22 \% \pm 4.7\) of Iowa’s democrats will support Warren.” But keep in mind, 22% is still the most likely outcome given our sample!
I want to emphasize this. You can see from the graph that 22% isn’t likely to be correct. Technically, the probability of any point in a continuous distribution is 0%. We have no idea in which direction the true population support is actually in; there’s an equal chance that it’s above or below 22%.
Here are the full results of the poll:
Here are three possible ways to report these results:
“Warren is polling ahead of Klobuchar.”
“Warren is polling ahead of Biden.”
“Warren and Biden are in a statistical tie.”
\[ 1.04 * \sqrt{\frac{(.5)(.5)}{439}} \] \[ = 2.5\% \]
I agree with Daniel that it is wise for the media to mention margin of error, especially when the race is a close one.
The Times/Sienna College poll that I’ve been using throughout this post was used in a nytimes piece a couple of days after the poll was conducted. Here’s how they chose to visualize the results:
This figure was one of the first things you read in the article. I think its format is the norm for poll visualizations: percentages. I would prefer to see the type of graph that I created above: percentages with error bars. The margin of error isn’t even mentioned until about halfway down the article, and it’s never interpreted.
Honestly, I don’t know the extent to which voters are mislead by an article like this, and I don’t know the extent to which polls effect voting behavior. Well, I do know that there can be a bandwagon effect that makes poll results a bit self-fulfilling. But those questions are beasts of which I won’t take on here.
The bottom line is: let’s all be data literate around election time, and remember what the margin of error means!
If different polls are reporting different results, and the difference is greater than the sum of their respective margins of error, this difference is probably not due to chance. The difference is probably structural, and has to do with who they sampled. Here are some of the more consequential variations in polling methods:
Polling method. Pollsters select respondents by either:
House effect. This is the systematic over-representation of Democratic or Republican respondents. Here are a couple of examples:
The USC/L.A. Times polls for state races leaned democrat in 2010 because they used bilingual interviewers
Many polls over-estimated Clinton’s support in 2016 because college graduates, who by 2016 were generally left of the aisle, are more likely to take surveys
Weights. The above problems can theoretically be corrected by applying demographic weights, but this requires making assumptions that could be faulty.
Before I got into my thoughts on how margin of error should be reported, I want to review what margin of error really is (as painlessly as I can).
So the formula for the margin of error is as follows: \[ z * \sqrt{\frac{(p)(1-p)}{n}} \]
To solve this, we really just need to know what n (sample size) is. That’s because there are some standard practices in the polling world, that turn the formula into this:
\[ = 1.96 * \sqrt{\frac{(.5)(.5)}{n}} \]
Why?
The proportion of support p = .5: Pollsters will assume that any candidate’s chances are a toss-up (this, I just learned). 50-50 odds represents the upper bound of uncertainty, and yields the maximum margin of error. Basically, they use 50-50 to be conservative in their estimation.
Confidence level indicator z = 1.96: It’s conventional to use a 95% confidence level, so z = 1.96. Look up z scores if you want to understand why. You could use a larger confidence level, say 99%, but then your margin of error will be larger.
Number of respondents n: This what pollsters can really control. All else equal, more respondents will lower the margin of error. The tricky thing with polling is that you want your sample to be demographically representative of the voting population. Also, you want to capture likely voters, because only about 60% of elegible voters actually do so.