Return to Main Page | Ragout: Lancet Quiz: The Answer
A Spicy Stew of Economics, Politics, Data, Food, Carpentry, etc.
Thursday, June 23, 2005

Lancet Quiz: The Answer

Someone has finally attempted my quiz! Commenter Kevin, writes:
My guess is that Apfelroth's reasoning is along these lines: in a war, refugees move away from potential flash-points, so the population is not geographically disributed as it would be for a census. So if your sampling method derives from census data, you will be over-sampling violent areas.
No, that's not it. Kevin's critique has previously been raised (very mildly) by Daniel Davies, one of the Lancet study's defenders. But that isn't what Apfelroth is saying, and in my opinion the reliance on old Iraqi census data is probably only a small problem with the Lancet study.

The problem is this. Within cities and villages, the Lancet researchers drew a rectangle around the village on a map, divided the rectangle up into a grid of hundred meter squares, and chose one of the squares at random. Each square had an equal chance of being chosen even though they might have had vastly unequal populations.

Apfelroth, who seems to be knowledgable about surveys, knows that if you draw a random sample of neighborhoods with unequal populations, you have to re-weight the sample to take that into account. Apfelroth is saying that given the available data, this would have been very hard: "it seems quite likely that the grid rectangles created by driving around in a war zone were much smaller than the original census tracts used in the 'cumulative population lists'."

In fact, Apfelroth is being overly generous here, assuming that the Lancet researchers did their best with this crucial step (weighting for neighborhood population). Either he is giving them the benefit of the doubt, or he's failed to notice that the Lancet researchers did not take this vital step at all: they weight all neighborhoods (100m grid squares) equally. Because they weight neighborhoods with different populations equally, their sample is much more likely to choose people from low-population neighborhoods: neighborhoods near parks or rivers, neighborhoods on the fringe of the city.

Anyone who's ever seen a Red/Blue map of Presidential voting in US counties will understand this phenomenon. Bush won the vast majority of the counties in the US (something like 80%) covering the vast majority of the country's land area (maybe 95%, I'd guess). Suppose you took a poll using the Lancet grid method. First you draw a rectangle around the continental US, then you choose grid squares within the rectangle at random, and finally you survey 30 people in each selected grid square. You'd end up with a lot of grid squares with no one living in them at all (e.g., oceans and deserts). And the vast majority of people interviewed would be in the rural, "Red" areas. A survey like this would likely find that 95% of respondents voted for Bush even though only 51% of the country actually did.

Now the Lancet study isn't quite this bad. It was only after they'd chosen a set of cities, towns, and villages to survey, using a reasonable method that gave a larger chance of selection to larger towns, that they began drawing rectangles and grids, and sampling land-area rather than people.

This is a huge problem for the Lancet survey if fighting is heavier on the fringes of cities than in the center, as I think it is. It will cause an overestimate of the death toll (I think a large overestimate). My commenter, Kevin, thinks that fighting is more common in city centers. If he's right, the Lancet study is still biased, although towards finding too small a death toll. It's hard to be sure who's right about where most fighting has occurred, but I'll make my case in a later post.

Number 1 in Ragout Economics!

March 2004 / April 2004 / May 2004 / June 2004 / July 2004 / August 2004 / September 2004 / October 2004 / November 2004 / December 2004 / January 2005 / April 2005 / May 2005 / June 2005 / July 2005 / August 2005 / September 2005 / October 2005 /

First Team
Angry Bear
Crooked Timber
Brad DeLong
Economist's View
Mark Kleiman
Nathan Newman
Political Animal
Max Sawicky
Brian Setser
Sock Thief
Talking Points Memo
Matthew Yglesias

Second Opinion
Stephen Bainbridge
Marginal Revolution
Andrew Samwick
The Volokh Conspiracy

Third Way

Fourth Estate
Economic Reporting Review
New York Times
Washington Post

Fifth Republic
Le Figaro
Le Monde

Sixth Sense
The Intersection
In the Pipeline
What's New

Politics & Polls
Daily Kos
Donkey Rising
Electoral Vote Predictor
Rasmussen Tracking Polls

Art Sucks
Enzo Titolo
L’esprit d’escalier
A Level Gaze
Approximately Perfect

ragoutchef at yahoo dot com


Powered by Blogger