Thursday, September 21, 2017

The margin of stupid

Every so often, I see a news story or tweet hyping the fact that a modest but non-negligible percent of Americans said some crazy or horrible thing in a survey. Here are two examples:

The most chilling findings, however, involved how students think repugnant speech should be dealt with...It gets even worse. Respondents were also asked if it would be acceptable for a student group to use violence to prevent that same controversial speaker from talking. Here, 19 percent said yes. 

Update: It turns out this particular survey was a badly designed piece of crap. I'm not particularly surprised...

Racial slurs that have cropped up chants, e-mails and white boards on America's college campuses have some people worried about whether the nation's diverse and fawned-over millennial generation is not as racially tolerant as might be expected. 


So, from these two examples -- both of them in the Washington Post -- I'm supposed to believe that Millennials are a bunch of unreconstructed racists, except for the ones who go to college, who are a pack of intolerant leftists. 

It seems to me like there's something inherently suspicious about judging a group of people based on sentiments expressed by only 15 or 20 percent of those people. But beyond that, there's another problem here - the problem of whether we can really trust these surveys. 

Surveys give a false sense of precision, by reporting a "margin of error" (confidence interval). But that confidence interval comes purely from the fact that the sample is finite. It does not capture systematic error, like selection bias (Are the people who answer this survey representative of the population being sampled?). And it definitely doesn't capture the errors people themselves make when responding to surveys.

When I did happiness survey research with Miles Kimball, there was always the nagging question of whether people are really able to know how happy they are. Of course, the whole question of what "happiness" should mean is a difficult one, but presumably there are some neurochemical tests you could do to determine how good someone feels, at least relative to how they felt in the past. How well do survey responses reflect this "true" emotion? Do people in different countries have cultural pressures that make them respond differently? Do Americans feel the need to say they're happy all the time, while British people would be ashamed to admit happiness? And are people measuring their happiness relative to yesterday, or to their youth, or to how happy they think they ought to be?

These errors were things that we lumped into something we called "response style" (psychologists call it response bias). It's very very hard to observe response style. But I'd say we can make a pretty good guess that Americans - and possibly everyone - do a lot of random responding when it comes to these sorts of surveys.

For example, a 2014 survey reported that 26 percent of Americans said that the sun goes around the Earth. 

Now, maybe there are a bunch of pre-Copernican geocentrists out there in America (there certainly are the flat-earthers!). Or maybe people just don't think very hard about how they answer these questions. Maybe some people are confused by the questions. Maybe some are trolling. 

Whatever the cause, it seems like you can get 20 to 25 percent of Americans to say any ridiculous thing imaginable. "Do you think eating raccoon poop reduces the risk of brain cancer?" "23 percent of Americans say yes!" "Would you be willing to cut your toes off with a rotary saw if it meant your neighbor had to do the same?" "17 percent of Americans say they would!" Etc.

You can also see this just from looking at some of the crosstabs in the first survey above. 20 percent of Democrats and 22% of Republicans say it's OK to use violence to shut down speakers you don't like. This sounds kind of nuts, given the panic on the right over lefty violence against campus speakers. Why would Republicans even more likely than Democrats to condone this sort of violence? It makes no sense at all...unless you can get ~20 percent of Americans to say pretty much any ridiculous thing on a survey. 

I call this the margin of stupid. Unlike the margin of error, it's not even a roughly symmetric error -- because you can't have less than 0% of people give a certain answer on a survey, the margin of stupid always biases surveys toward showing some non-negligible amount of support for any crazy or stupid or horrible position. 

Whenever you read a survey like this, you must take the margin of stupid into account. Yes, there are Americans who believe crazy, stupid, and horrible things. But dammit, there aren't that many. Next time you see some poll breathlessly claiming that 21 percent of Americans support executing anyone whose name starts with "G", or that 18 percent of Millennials believe themselves to be the reincarnation of Kublai Khan, take it with a grain of salt. It's a lot easier to give a stupid answer on a survey than to actually truly hold a nuts belief.

Sadly, the margin of stupid also probably applies to voting.


  1. Anonymous2:13 AM

    Scott Alexander called it the Lizardman's Constant, based on a survey that found that 4% of Americans believe that lizardmen secretly control the government:

    So the margin of stupid probably isn't as high as 15-20%, more like 4-5%.

    1. Well that is the really really stupid.

      I'm sure there is a slightly larger bracket in the kinda really stupid category.

  2. "It seems to me like there's something inherently suspicious about judging a group of people based on sentiments expressed by only 15 or 20 percent of those people."

    I agree. Yet I still get lumped into being a racist, xenophobic, misogynist all the time because I'm on the right. And you go along with it plenty Noah!

  3. Does anyone want to study this? i.e. do a bunch of methodologically rigorous polling asking stupid things and seeing how many people will say yes. I seem to remember in the UK that when doing drug taking surveys they include a fake drug and exclude the results of anyone who says they took it.

  4. It's called the Crazification Factor.

  5. Anonymous11:03 AM

    Scott Alexander calls it the Lizardman's Constant.

  6. Re: Lizardmen, the "margin of stupidity" isn't fixed but depends on the question's ease of interpretation. The moment you say "lizardmen," unless the question uses some kind of tortured syntax, it's obvious that the answer is "no," I don't believe that. But look at the specific wording of the "campus speaker" question. It's long and compound and features three different actors (the university, the speaker, the student group) and then asks whether you "agree or disagree" with one of them. Forget about "stupid," this is the kind of question a perfectly sensible person can easily misunderstand.

  7. Two choices, half answering at random, 25%. Random vs perversely would be interesting, such as asking the same question in two different forms.

  8. Anonymous3:03 PM

    I think with these types of surveys it is very hard to ascertain exactly what the question is asking. I believe it is reasonable to think that violence to shut down certain forms of speech is philosophically permissible but usually inadvisable, Certainly at the very least I can sit in my armchair and imagine speech which could warrant such a strong reaction. In these circumstances I never know if I am being asked the philosophical question of whether it is conceivably permissible or the political/social question of whether it should be an allowed norm.

  9. Anonymous4:27 PM

    Ask a stupid question and you'll get a stupid answer.

  10. Anonymous9:26 AM

    How about "bad survey design"?