The Squid Strategy
When you're wrong, and someone points it out, the squid strategy is a good one. Fill the water with black ink to confuse the issue, and hope that observers will through up their hands and decide that it's all too complicated, and too much trouble to judge who's right.
This kind of thing is pretty common in politics, but via
Brad DeLong, I learn how Harvard Economist Caroline Minter Hoxby is doing the same thing. Hoxby wrote a well-known paper arguing that competition, in the form of numerous school districts, improves school productivity and student outcomes.
In fact, relatively simple statistical methods find no such relationship, but Hoxby argues that there might be an omitted variable or reverse causation problem. For example, maybe good school districts get bigger, reducing competition. This doesn't seem like such a serious statistical problem to me, but Hoxby famously proposed to solve it using rivers as a "natural experiment." Her idea is that metropolitan areas with more rivers will have more school districts, because in the olden days, it was hard to cross rivers to go to school. To simplify a little, Hoxby shows that metro areas with more rivers have higher student achievement, and so concludes that competition is a good thing.
Recently, the tenured Hoxby has been criticized by the untenured Jesse Rothstein of Princeton, who says he found numerous errors in her study.
Hoxby responds with the squid defense, rebutting Rothstein's weaker and more technical points, while not providing a convincing answer to his main criticism. An economist at the Lowest Deep blog is taken in, writing: "Rothstein's criticisms are lengthy, technical, and enumerated in excruciating detail." So is Brad DeLong, writing "I don't have a dog in this fight. I'm not qualified to judge this fight."
Oh! It's all so complicated! How can a layperson judge! Or even an economist in another subfield! Well let me explain these deeply technical and complicated matters.
The whole thing hinges on the definition of rivers. Hoxby says that it's small rivers that determine how many school districts are formed in a metropolitan area, not big rivers. Rothstein pretty convincingly shows that Hoxby's provides no clear definition of small rivers, and that her decisions about what to count are extremely subjective. When you replace Hoxby's measure of "small rivers" with a simpler measure of the total number of rivers in a metro area, Hoxby's results disappear.
As far as I can tell, Hoxby doesn't dispute this point. She has a vague and unconvincing argument that her count of small rivers is the right one, but mostly tries to shift attention to more technical matters.
Rothstein's
main point is summarized pretty well in his abstract
In an influential paper, Hoxby (2000) studies the relationship between the degree of so-called "Tiebout choice" among local school districts within a metropolitan area and average test scores. She argues that choice is endogenous to school quality, and instruments with the number of larger and smaller streams. She finds a large positive effect of choice on test scores, which she interprets as evidence that school choice induces greater school productivity. This paper revisits Hoxby's analysis. I document several important errors in Hoxby's data and code. I also demonstrate that the estimated choice effect is extremely sensitive to the way that "larger streams" are coded. When Hoxby's hand count of larger streams is replaced with any of several alternative, easily replicable measures, there is no significant difference between IV and OLS, each of which indicates a choice effect near zero. There is thus little evidence that schools respond to Tiebout competition by raising productivity.
See, not really all that complicated. It's all a matter of which streams count as large and which as small. The depressing thing is that Hoxby is a better writer than Rothstein, so I bet she's going to win the debate.
|
