When Confidence Intervals Overlap
Whenever I write about polls, I'm reminded of how tricky it is to interpret the margin of error. In particular, I never see a distinction made between the confidence interval for a politician's lead in the polls, and a politician's gain over time. But they're not the same. Not only that, but neither are what the newspaper reports as the confidence interval. Although it's tempting to bash the press, professional pollsters get this wrong too.
Suppose we take a poll of 600 people, and Kerry gets 53 to Bush's 47. Newspapers would report this as having a "margin of error" of +/- 4. (So, I'm thinking about a poll smaller than the 1000 person polls, with a margin of error of +/- 3, that we usually see reported.) The margin of error of 4 means means that Kerry's 95% confidence interval is 53 +/- 4, or between 49 and 57.
The confidence intervals overlap, so this is a "statistical tie."* To say this another way, the confidence interval for the difference is 8, so Kerry is ahead by 6 +/- 8: there's a 95% chance that he's somewhere between 14 points ahead and 2 points behind. The margin of error for Kerry's lead (+/- 8), is double that for his percentage of the vote (+/- 4).
Suppose we take another poll a week later, again with 600 people, and the results are reversed: Kerry gets 47 to Bush's 53, again +/-4. Can we say that Bush has gained? Yes, Bush's gain from 47 to 53 is outside the margin of error, even though the confidence intervals overlap
. In fact, the confidence interval for the gain from last week is 6 +/- 5.7. The 5.7 percent margin of error is calculated as 1.4 (the square root of two) times 4 (the margin of error for Bush's percentage of the vote).
Why the difference? Because in a single poll of Bush vs. Kerry, the responses aren't independent. If the pollsters happen to speak with a sample that contains an unusually high number of Kerry supporters, by definition the sample will also have an unusually low number of Bush supporters. This negative correlation between Bush's and Kerry's percentage will tend to increase the difference between the two candidates. However two separate polls, taken a week apart, will be independent. If one sample overestimates Kerry's support by chance, there's no reason to think that next week's sample will also tend to overestimate or underestimate it.
Here's a good explanation
for the case where we're calculating a politician's lead at a point in time. Here's a respected polling organization that gets it wrong
. They use the formula for a gain over time, when they should be using the formula for a lead at a point in time.
*. If Kerry gets 53 to Bush's 47, it's obviously more likely that Kerry is ahead than behind. The "statistical tie" language is somewhat misleading. For example, you rarely see an 80% confidence interval reported, but it's 53 +/- 2.6, so with 80% confidence we can say that Kerry's ahead, even though we can't say that with 95% confidence.