Academic racism has a K=N problem

Let's use the term "academic racism" to mean "“a belief that race is the primary determinant of human traits and capacities and that racial differences produce an inherent superiority of a particular race" (the Merriam-Webster "full definition" of "racism"), in order to differentiate it from bigotry (the common-use definition).

Anyway, science writer Nicholas Wade has a new book out making the standard case for academic racism. Andrew Gelman, a statistician, has a review of that book in Slate. The review is good, and you should read it, but I thought I'd try to restate Gelman's point in a slightly more compact way.

Basically, academic racism has a problem, and that problem is overfitting.

Here's how academic racism generally works. Suppose you see two groups that have an observable difference: for example, suppose you note that Hungary has a higher per capita income than Romania. Now you have a data point. To explain that data point, you come up with a theory: the Hungarian race is more industrious than the Romanian race. But suppose you notice that Romanians generally do better at gymnastics than Hungarians. To explain that second data point, you come up with a new piece of theory: The Romanian race must have some genes for gymnastics that the Hungarian race lacks.

You can keep doing this. Any time you see different average outcomes between two different groups, you can assume that there is a genetic basis for the difference. You can also tell "just-so stories" to back up each new assumption - for example, you might talk about how Hungarians are descended from steppe nomads who had to be industrious to survive, etc. etc. As new data arrive, you make more assumptions and more stories to explain them. Irish people used to be poor and are now rich? They must have been breeding for richness genes! Korea used to be poorer than Japan and is now just as rich? Their genes must be more suited to the modern economy! For every racial outcome, there is a just-so story about why it happened. Read an academic-racist blog, like Steve Sailer's, and you will very quickly see that this kind of thinking is pervasive and rampant.

There's just one little problem with this strategy. Each new assumption that you make adds a parameter to your model. You're overfitting the data - building a theory that can explain everything but predict nothing. Another way to put this is that your model has a "K=N" problem - the number of parameters in your model is equal to the number of observations. If you use some sort of goodness-of-fit criterion that penalizes you for adding more parameters, you'll find that your model is useless (no matter how true or false it happens to be!). This is one form of a more general scientific error known as "testing hypotheses suggested by the data", or "post-hoc reasoning". It's a mistake that is by no means unique to academic racism, but instead is common in many scientific disciplines (cough cough, sociobiology, cough cough).

Gelman explains all this in layman's terms.

None of this is to say that academic racism is completely wrong, or is a useless way of looking at the world! It might get certain things right. But we don't really know (yet). We don't have a good understanding of the proposed causal mechanism, and we don't have good natural experiments to test any of the hypotheses. Suppose academic racism gets half of its hypotheses right (suppose Romanians really do have a "gymnastics gene" that Hungarians lack), but gets half of them wrong (suppose Hungarians have higher per capita GDP because of better institutions and proximity to German markets, not because of an "industriousness gene"). How would we tell the difference between the right parts and the wrong parts? 

We couldn't.

Academic racism is very alluring, for at least three reasons. First, it tells us that all our stereotypes and prejudices are basically right - and we humans like to be told that all our preconceptions are right. We suffer from confirmation bias. Second, academic racism feels cool and edgy and rebellious, because political correctness still often banishes it from the realm of acceptable discourse. It's fun to feel like the scientific rebel, fighting for The Facts against the thought control of The Establishment. And third, academic racism provides a convenient excuse for racism of the non-academic kind. Scared that a big, masculine black guy will take your girlfriend? Worried that hard-working, intelligent (but "uncreative") immigrants will take your job? Academic racism provides convenient stories to justify policies that protect you against threats like these - at the expense of the black guys and the immigrants, of course.

This is why even though academic racism could be right about some stuff, I just roll my eyes whenever I see it. "Race" is a kind of phlogiston that can be invoked to explain pretty much anything (kind of like "culture" and "technology" and "institutions" in economics!). Until we have a much better understanding of genetics, the infinitely proliferating hypotheses of academic racism will be neither proven nor disproven, and people will go on believing in them (or disbelieving them). The theory that Jeopardy champion Arthur Chu called a "maggoty corpse" will shamble onward, never dying, never truly alive.