Recently, my friend Daniel Stublen made the argument that reporters should always include the margin of error when citing polls. This led to a good discussion about what the margin of error really is, but I wanted to dive deeper into it, here.

Margin of Error

Polls are bound to have sampling error (the error that occurs because you ask a subset of the population, rather than the entire population). The margin of error measures the maximum amount by which the sample results are expected to differ from those of the actual population. The lower the margin, the more confident you can be in your estimation.


Let’s consider a real-world example. Between October 25-30, 2019, the New York Times Upshot/Siena College asked 439 Iowans “Who is your first choice?” and 22% responded with Elizabeth Warren. The margin of error on this was 4.7 percentage points.

So the interpretation of this poll is: “At least 95 out of 100 times, \(22 \% \pm 4.7\) of Iowa’s democrats will support Warren.” But keep in mind, 22% is still the most likely outcome given our sample!

I want to emphasize this. You can see from the graph that 22% isn’t likely to be correct. Technically, the probability of any point in a continuous distribution is 0%. We have no idea in which direction the true population support is actually in; there’s an equal chance that it’s above or below 22%.

How to Report the Horse Race

Here are the full results of the poll:

Here are three possible ways to report these results:

“Warren is polling ahead of Klobuchar.”

  • I think this is totally okay to say, without caveats. Because the difference in the two candidates’ support is greater than their margins (greater than a 9.2-point lead, in this case), Warren’s lead is probably not due to sampling error.

“Warren is polling ahead of Biden.”

  • Eh, I think this is sketchy. Yes, Warren is polling ahead, but that lead could be due to sampling error.

“Warren and Biden are in a statistical tie.”

  • I think this take is an exaggeration. Sure, her lead on him is less than 9.2 points, but that number comes from the 95% confidence level. What if we use a 70% confidence level? Then, using the corresponding z-score, our margin of error would be…

\[ 1.04 * \sqrt{\frac{(.5)(.5)}{439}} \] \[ = 2.5\% \]

       …and the graph would look more like this:

       Now, with 70% confidence, Warren’s lead over Biden is        significant.

The Verdict

I agree with Daniel that it is wise for the media to mention margin of error, especially when the race is a close one.

The Times/Sienna College poll that I’ve been using throughout this post was used in a nytimes piece a couple of days after the poll was conducted. Here’s how they chose to visualize the results:

This figure was one of the first things you read in the article. I think its format is the norm for poll visualizations: percentages. I would prefer to see the type of graph that I created above: percentages with error bars. The margin of error isn’t even mentioned until about halfway down the article, and it’s never interpreted.

Honestly, I don’t know the extent to which voters are mislead by an article like this, and I don’t know the extent to which polls effect voting behavior. Well, I do know that there can be a bandwagon effect that makes poll results a bit self-fulfilling. But those questions are beasts of which I won’t take on here.

The bottom line is: let’s all be data literate around election time, and remember what the margin of error means!

Why do pollsters get different numbers?

If different polls are reporting different results, and the difference is greater than the sum of their respective margins of error, this difference is probably not due to chance. The difference is probably structural, and has to do with who they sampled. Here are some of the more consequential variations in polling methods:

Polling method. Pollsters select respondents by either:

  1. Calling a list of registered voters: This captures active voters, but leaves out people who don’t put their phone number on their voter registration, as well as the unregistered new voters.
  2. Random digit dialing: This captures a larger universe, but it relies on respondents to be truthful about their registration status.
  3. Using online polls: This… is cheap. Phone response rates are pretty low these days at 6%, which means more dialing, which means surveys get costlier. Also, respondents are more likely to be truthful online, so this method avoids the Bradley effect. However, being a nascent method, their accuracy is questionable.

House effect. This is the systematic over-representation of Democratic or Republican respondents. Here are a couple of examples:

Weights. The above problems can theoretically be corrected by applying demographic weights, but this requires making assumptions that could be faulty.

The Formula (if you care)

Before I got into my thoughts on how margin of error should be reported, I want to review what margin of error really is (as painlessly as I can).

So the formula for the margin of error is as follows: \[ z * \sqrt{\frac{(p)(1-p)}{n}} \]

To solve this, we really just need to know what n (sample size) is. That’s because there are some standard practices in the polling world, that turn the formula into this:

\[ = 1.96 * \sqrt{\frac{(.5)(.5)}{n}} \]

Why?

  • The proportion of support p = .5: Pollsters will assume that any candidate’s chances are a toss-up (this, I just learned). 50-50 odds represents the upper bound of uncertainty, and yields the maximum margin of error. Basically, they use 50-50 to be conservative in their estimation.

  • Confidence level indicator z = 1.96: It’s conventional to use a 95% confidence level, so z = 1.96. Look up z scores if you want to understand why. You could use a larger confidence level, say 99%, but then your margin of error will be larger.

  • Number of respondents n: This what pollsters can really control. All else equal, more respondents will lower the margin of error. The tricky thing with polling is that you want your sample to be demographically representative of the voting population. Also, you want to capture likely voters, because only about 60% of elegible voters actually do so.