Industrial statistician George Box is credited for the saying that “all models are wrong, but some are useful.” Andrew Gelman, professor of statistics at Columbia, shares some thoughts on the saying:
With a small sample size, you won’t be able to reject even a silly model, and with a huge sample size, you’ll be able to reject any statistical model you might possibly want to use.
I wasn’t familiar with this quote, but the discussion is interesting. I think the following comment made by Gelman captures the point of the quote well (although, admittedly, Gelman is specifically talking about model checking, but this distilled version works):
[The point is] to understand what aspects of the data are captured by the model and what aspects are not.
Gelman’s comments come after reading a rant (or personal essay) by professor of statistics J. Michael Steele at Wharton. I particularly like Steele’s discussion of “fitness for purpose”. As he says:
The majority of published statistical methods hunger for one honest example.
Steele highlights the shortcomings of model adequacy and provides links to a couple of short notes that take the discussion further. Does the model make sense? and Does the model make sense? Part II: Exploiting sufficiency. At this point the discussion can explode into articles and chapters from Gelman and Buja, which Steele refers to directly, among others.
I find Steele to be a very practical, down-to-earth sort. I’m going to have to keep an eye on his writing. I only wish he had a blog we could follow.
Lies and Stats 