Return to Main Page | Ragout: When Confidence Intervals Overlap
A Spicy Stew of Economics, Politics, Data, Food, Carpentry, etc.
Tuesday, August 03, 2004

When Confidence Intervals Overlap

Whenever I write about polls, I'm reminded of how tricky it is to interpret the margin of error. In particular, I never see a distinction made between the confidence interval for a politician's lead in the polls, and a politician's gain over time. But they're not the same. Not only that, but neither are what the newspaper reports as the confidence interval. Although it's tempting to bash the press, professional pollsters get this wrong too.

Suppose we take a poll of 600 people, and Kerry gets 53 to Bush's 47. Newspapers would report this as having a "margin of error" of +/- 4. (So, I'm thinking about a poll smaller than the 1000 person polls, with a margin of error of +/- 3, that we usually see reported.) The margin of error of 4 means means that Kerry's 95% confidence interval is 53 +/- 4, or between 49 and 57.

The confidence intervals overlap, so this is a "statistical tie."* To say this another way, the confidence interval for the difference is 8, so Kerry is ahead by 6 +/- 8: there's a 95% chance that he's somewhere between 14 points ahead and 2 points behind. The margin of error for Kerry's lead (+/- 8), is double that for his percentage of the vote (+/- 4).

Suppose we take another poll a week later, again with 600 people, and the results are reversed: Kerry gets 47 to Bush's 53, again +/-4. Can we say that Bush has gained? Yes, Bush's gain from 47 to 53 is outside the margin of error, even though the confidence intervals overlap. In fact, the confidence interval for the gain from last week is 6 +/- 5.7. The 5.7 percent margin of error is calculated as 1.4 (the square root of two) times 4 (the margin of error for Bush's percentage of the vote).

Why the difference? Because in a single poll of Bush vs. Kerry, the responses aren't independent. If the pollsters happen to speak with a sample that contains an unusually high number of Kerry supporters, by definition the sample will also have an unusually low number of Bush supporters. This negative correlation between Bush's and Kerry's percentage will tend to increase the difference between the two candidates. However two separate polls, taken a week apart, will be independent. If one sample overestimates Kerry's support by chance, there's no reason to think that next week's sample will also tend to overestimate or underestimate it.

Here's a good explanation for the case where we're calculating a politician's lead at a point in time. Here's a respected polling organization that gets it wrong. They use the formula for a gain over time, when they should be using the formula for a lead at a point in time.

*. If Kerry gets 53 to Bush's 47, it's obviously more likely that Kerry is ahead than behind. The "statistical tie" language is somewhat misleading. For example, you rarely see an 80% confidence interval reported, but it's 53 +/- 2.6, so with 80% confidence we can say that Kerry's ahead, even though we can't say that with 95% confidence.

Number 1 in Ragout Economics!

March 2004 / April 2004 / May 2004 / June 2004 / July 2004 / August 2004 / September 2004 / October 2004 / November 2004 / December 2004 / January 2005 / April 2005 / May 2005 / June 2005 / July 2005 / August 2005 / September 2005 / October 2005 /

First Team
Angry Bear
Crooked Timber
Brad DeLong
Economist's View
Mark Kleiman
Nathan Newman
Political Animal
Max Sawicky
Brian Setser
Sock Thief
Talking Points Memo
Matthew Yglesias

Second Opinion
Stephen Bainbridge
Marginal Revolution
Andrew Samwick
The Volokh Conspiracy

Third Way

Fourth Estate
Economic Reporting Review
New York Times
Washington Post

Fifth Republic
Le Figaro
Le Monde

Sixth Sense
The Intersection
In the Pipeline
What's New

Politics & Polls
Daily Kos
Donkey Rising
Electoral Vote Predictor
Rasmussen Tracking Polls

Art Sucks
Enzo Titolo
L’esprit d’escalier
A Level Gaze
Approximately Perfect

ragoutchef at yahoo dot com


Powered by Blogger