Prominent academics are often astonished at the rapidity with which the blogosphere occasionally pounces on and dissects their research findings. In this case, it happened to Case and Deaton, authors of a recent much-publicized study entitled "Rising morbidity and mortality in midlife among white non-Hispanic Americans in the 21st century." The pounce was done by Phil Cohen, and - most prominently - by Andrew Gelman.
The TL;DR version is that rising mortality in some of the subgroups spotlighted by Case and Deaton was increased by a composition effect - the average age within the subgroups increased over the observation period, which pushed up death rates for the aggregated subgroups. If you remove the composition effect, the mortality increase among these groups was considerably less.
Anne Case responded with the consternation typical of researchers first encountering blog attacks:
Case said that she didn’t buy this argument. “We spent a year working on this paper, sweating out every number, sweating out over what we were doing, and then to see people blogging about it in real time — that's not the way science really gets done,” she said. “And so it’s a little hard for us to respond to all of the blog posts that are coming out.”
Academics are used to the cozy, staid world of academia. Responses are slow, polite, and vetted by third parties. Arguments happen in seminars, in office discussions, and at dinners. Disputes are resolved over a matter of years - when they are resolved at all. And never do intellectual adversaries take their case to the general public!
But academics are going to have to get used to blogs. The technological advances of the web have simply made it easier for crowds of outsiders to evaluate research in real time. How often that process produces the "wisdom of crowds", and how often it merely adds unhelpful noise, remains to be seen. Certainly we've seen the internet do both of those things at different times. But blog criticism of research looks like something that's here to stay, and academics whose work appears in the popular press will have to get used to it!
Blog discourse has some distinct advantages - above all, the speed of responses and the diversity of people who get involved in discussions. How often do you see two economists arguing with a sociologist and a political scientist/statistician? That's pretty cool! There is, however, a tendency for blog debates to become too antagonistic.
I think Andrew Gelman's latest salvo against Case and Deaton falls into this category a bit. He is put out that Case and Deaton have, so far, refused to issue a public mea culpa about what he sees as a major gotcha. Gelman writes up what he thinks such a mea culpa should say, and includes these bits of snark:
Had it not been for bloggers, we’d still be in the awkward situation of people trying to trying to explain an increase in death rates which isn’t actually happening...We count ourselves lucky to live in an era in which mistakes can be corrected rapidly[.]
Gelman is dramatically overstating the importance of what he found! To say that the increase in death rates "isn't actually happening", first of all, is not quite right - Gelman's rough-and-ready composition adjustment removes all of the increase, but more careful examination shows that some portion of the increase remains.
Second, Gelman is kind of assuming that zero is the important benchmark for what constitutes an "increase". He makes sure to point out that the paper's main finding - that American white mortality increased a lot relative to various comparison groups - is not changed by the composition adjustment. But when he claims that the increase "didn't really happen", Gelman is saying that "increase" is an absolute rather than a relative term.
Andrew, you're a stats guy. You know full well that people analyzing time-series data detrend stuff all the time. Measuring increases relative to a trend is totally standard practice!
So like many blog debates, this one ends up making a mountain out of a molehill. The composition effect was a useful and instructive observation, but it doesn't really change anything about the paper's result. And publicly demanding that the authors engage in an equally public mea culpa over such a non-issue is a little unrealistic. If it leads to rancor in the long term, that will be a shame.
I like what blogs have done for research, but I think we should work to make those discussions less about point-scoring and more about a cooperative, crowdsourced search for truth.
This is reminding me of Tyler Cowen's point that the internet has disproportionately improved the lives of infovores*. I'm a big nerd but I didn't go into academia. Thanks to twitter and RSS feeds however, I got to:
ReplyDelete* see Andrew Gelman share his post on twitter
* read it, and find it a tad aggressive
* have this come up in my RSS feed to provide confirmation and further commentary
Great for nerds, while slightly reducing the need to go to grad school.
*Infovore is just a term Tyler made up as an alternative to nerd, right?
Hi, Noah. Thanks for the post. I just have a few comments:
ReplyDelete1. Your title is "academics vs. blogs." But I am an academic too. Blogging is one of the ways that I work out and communicate ideas.
2. You describe my post as snarky. On the contrary, it is written in complete sincerity.
3. When the age-aggregation bias is removed, there is no increase in death rates among this group since 2005. The data show a decrease from 1989-1999, an increase from 1999-2005, and a flat pattern since 2005 (a mix of an increase for women and an increase for men). If you read the many pieces of punditry on the topic, you see lots of explanation for an increase in death rates in this group, but, no, this increase is not happening. 2005 was ten years ago.
4. It happens from time to time that people find errors in my work and they point out these errors to me. When this happens, I thank them. It's really not so hard to do. I never asked (let alone "demanded") that Case or Deaton engage in a "mea culpa." I think a Thank you from them would be appropriate, but what really upset me was not any lack of thanks but rather that they aggressively criticized the corrections that I and others did. We did science, and their response was that "that’s not the way science really gets done." That's not an attitude that's conducive to a cooperative, crowdsourced search for truth.
Your title is "academics vs. blogs." But I am an academic too. Blogging is one of the ways that I work out and communicate ideas.
DeleteSure.
You describe my post as snarky. On the contrary, it is written in complete sincerity.
Once upon a time, I was surprised to discover that what other people called "snark" was my natural mode of interaction... ;-)
I think a Thank you from them would be appropriate, but what really upset me was not any lack of thanks but rather that they aggressively criticized the corrections that I and others did. We did science, and their response was that "that’s not the way science really gets done." That's not an attitude that's conducive to a cooperative, crowdsourced search for truth.
Well you're right, of course, but I don't think it's something worth getting upset about...
Prof. Gelman seems unhappy about the response of Case and Deaton, because other writers have been citing their study in support of any number of hypotheses, based on a false interpretation of the results. For example, consider this gem:
Deletehttp://noahpinionblog.blogspot.com/2015/11/case-deaton-and-human-capital-debate.html
I quote: "This paper provides some hard data to corroborate a story we have been seeing elsewhere: College-educated Americans are significantly healthier in their personal, family, and social lives. To me this indicates that education has acted to partially innoculate Americans against the overall negative changes that are affecting our society."
What Prof. Gelman and others have been disputing is precisely that these data support the idea of "overall negative changes" that people need to be innoculated against. Instead, they show that most groups have seen declining mortality rates, and middle-aged White Americans declined a bit, rose a bit, and flatlined for the last decade (averaging out diverging trends in the gender subgroups). You can argue that their results remain interesting and important, but once corrected they render the linked analysis completely unsupported. The same goes for Douthat and Krugman's columns on the topic. If Deaton and Case were to concede the point, then perhaps such careless readings would be less widespread. Alternatively, they could argue against it. Instead, they're dismissing it, suggesting that their original description was A-OK and all the silly interpretations we see all over the web and in the media are consistent with their data. That's what is unfortunate.
Is it OK to point out that Gelman, in all this, has been hell-bent in pointing out that he IS NOT trying to dismiss Case and Deaton´s results when compared to other countries?
Delete"The TL;DR version is that rising mortality in some of the subgroups spotlighted by Case and Deaton was increased by a composition effect - the average age within the subgroups increased over the observation period, which pushed up death rates for the aggregated subgroups."
ReplyDelete-This is generally a non-issue. Are Germany and Canada not aging? The issue is the divergence of U.S. Whites from the rest, not any absolute increases or decreases in death rates.
As the academic above says, "still newsworthy."
"So like many blog debates, this one ends up making a mountain out of a molehill. The composition effect was a useful and instructive observation, but it doesn't really change anything about the paper's result."
-Indeed. I blame the U.S. healthcare system. The big mystery, however, which no one has mentioned, is Asians. They're not as poor and disconnected from the U.S. healthcare system as NAMs.
This hissyfit on both sides finally helps me understand what the heck happened at Yale. Crimony, grow up, people. How precious can tenure make you?
ReplyDeleteFor John, caring about ideas = precious hissyfits.
DeleteAnd re your later comment: are you really trying to say people in other walks of life don't get hung up on status?!
John:
ReplyDeleteAs a scientist, I don't like when someone describes what I do as "that's not the way science really gets done."
This has nothing to do with tenure or being precious or hissy or whatever. I don't like to be insulted, it's that simple. I work hard when I do my work. When someone finds problems with my work, I think it's great when they point out what I do wrong. But if someone can't actually point out anything I did wrong, damn straight I don't like it when they characterize my work as "not the way science really gets done."
Beyond any personal reaction, there's a larger point which is that self-appointed gatekeepers of science can hinder science in two ways: first by disparaging the work that they don't like (as noted above) and second by creating a false sense of security regarding the correctness of published work, which has been recently recognized to be a problem in many different fields, including psychology and economics. So, yeah, I xo think it's a problem when people engage in this sort of gatekeeping rather than embracing "a cooperative, crowdsourced search for truth," in Noah's words.
Andrew:
DeleteThank you for pointing out my opinion about your actions is wrong. Irony is not dead. You might be surprised how this whole teacup-affair strikes laymen, and the inability to do so might leave you confused as to why most others don't judge you by your internal thoughts rather than your actions.
It also is frankly discouraging to me that almost daily I watch academia get sidetracked from working to solve problems that could conceivably touch lives other than their own to devote themselves to status-maintenance.
I ramble. Shorter: the world is not about you. If you have time for this nonsense, we seem to have underemployed economists. Perhaps if you were a non-college educated middle-aged white male, you might have different priorities.
Hi, I wrote a long comment but I realized it was beside the point. If you think there's higher-prioriity work I should be doing, rather than responding to blog comments, you're probably right. I guess you and I and Anne Case and Noah Smith and everyone else should all go back to work.
DeleteGelman, I am going to hypothesize that your real problem here is that you come across as thin-skinned. It is possible to combine ego and insights in a way that works--Krugman is a master at this. Is he thin-skinned? He doesn't seem so to me, and he gets mad (rightly) when people take things out of context or continue to spout BS when evidence or logical contradictions suggest otherwise. But your posts have this think-skinnedness (if such a word exists) to them, and I wonder if this isn't turning people off.
DeleteI think it is important to note that IF the academics have thought everything out so carefully, and vetted so carefully, then they should, at least usually, be able to quickly answer quick criticism from the blogosphere. When you've really studied it and done it well, then frivolous or spurious criticism you should usually be able to reply to well in like 10 minutes, at least the basics.
ReplyDeleteI could, however, see a concern that your response won't get out nearly as much as the initial criticism, or your response may be correct but harder to understand.
I'll go further, if the public has spurious criticisms, just shutting up is probably not best. It seems much better for them to be able to ask, and then if the criticisms really are spurious, the authors should usually be able to without that much effort explain why, then the public learns. But the thing is, it's so much more than the general public when it includes people like Andrew Gelman -- full professor at an Ivy league school in statistics! And making a statistics point! He should be asking and be answered?
DeleteAnd, I'll add, just keeping the vetting to behind closed doors and in that sub-area only:
Delete1) Allows people in the field and gatekeepers to hush things that hurt their prestige and skills and, possibly, make their ideology look worse.
2) Doesn't expose it to nearly as many people, especially highly skilled people outside of the (perhaps very narrow) sub-field, like Gelman here.
To me the most surprising aspect of this episode was that these findings are new. My expectation was that there are hundreds of health researchers sitting at government offices who have these charts memorized to 3 significant digits. I mean, mortality is important, many people look at mortality data. If you have the data, slicing mortality rate by ethnicity is probably one of the first things you would do as a data analyst. What am I missing?
ReplyDeleteI'd like to second Daniel Varga's question!
DeleteDaniel,
DeleteAlthough I am being lazy and not providing a link, I remember seeing journalistic reports over recent years about how death rates were actually rising for poor white women in the US (West Virginia singled out in at least one story), with such things as smoking, drug use, obesity, and suiicide singled out as causes. It looks like Gelman's data suggests that whether the rate is rising or just flatlined for the entire age/racial group depends on whether falling rates for men balance out rising rates for women or not, the answer to which seems to be up in the air, although it looks like Case-Deaton overstated the overall rate of increase, whether or not it is positive.
So, Case-Deaton seem to have been the first academics to hit the journals with a study of this, and they reportedly were worried about being "scooped" by others when their paper was turned down initially at two journals. They do seem to have beaten the crowd of academics. But it is clear that there have been spme datamesters on this well aware of it for some time, given that it has been in the news media, although Case-Deaton made it much more of a front page story rather than an odd curiosity as it has been so far.
On the general fisticuffs, I generally agree with Noah's view here. I think Gelman needs to recognize that there are competing estimates to his, and that even if he is right that correcting for the composition effect leads to a flatlined outcome, this is still news, and I think it is sexist that there has been so little publicity about how it looks like the rate is rising for poor white women no mattter what. OTOH, it looks like Case has let Deaton's Nobel go to her head (they are married), and her dismissive remarks about criticism in blogs just makes her look petty, and, well, unscientific.
Barkley Rosser
Yes, I recall the paper(s) and data that "white men die faster than before" about ten years ago...so this is old news indeed, perhaps a small refinement of that research. Boring...
DeleteI think that there is at least one explanation that works for leveling off death rate that does not work for a rising death rate. So the IMHO the difference is very important.
ReplyDelete"But when he claims that the increase 'didn't really happen', Gelman is saying that 'increase' is an absolute rather than a relative term."
ReplyDeleteBut no one bothered to adjust the comparison groups for composition changes either. Many other countries had baby booms as well after WWII.
But when he claims that the increase "didn't really happen", Gelman is saying that "increase" is an absolute rather than a relative term.
ReplyDeleteAndrew, you're a stats guy. You know full well that people analyzing time-series data detrend stuff all the time. Measuring increases relative to a trend is totally standard practice!
Sure but, Noah, you're an econ guy. You know that the difference between a relative change and an absolute one is frequently incredibly important. "US GDP has decreased relative to Sweden" doesn't mean much of anything. "US GDP has decreased" means we're possibly heading into a recession.
Similarly, white mortality decreasing relative to Hispanic is not all that interesting. It wouldn't be surprising if Hispanic people's mortality decreased in the US for all kinds of very benign reasons. Absolute increases in white mortality would be a sign of something very bad.
“How often do you see two economists arguing with a sociologist and a political scientist/statistician?” Exactly. Economists have become quite arrogant in their refusal to look at evidence and analysis from other disciplines, especially sociology. Philip Cohen has a recent blog post along these lines here. Apparently, the only time economists will pay attention to non-economists is when the non-economists make a valid criticism of a paper that’s gotten attention in the non-academic media.
ReplyDeleteCase-Deaton made what looks like a rookie mistake that any demographer (Philip Cohen, for example) would have avoided. And even when the age-composition problem is pointed out, Case doesn’t address the issue itself but complains that the bloggers responded too quickly. (She also offers an argument that resembles that of the student complaining about a bad grade on a bad paper – “But I worked so hard on it.” I cringed when I read that sentence.)
Sociology is a discipline in name only, and to suggest it can provide evidence is the height of foolishness.
DeleteAnd Cohen in particular seems to be among the worst. His understanding of economics is lies only above yours and Robert Reich's.
DeleteBig Bad Bob, could you please post some of your copious evidence?
DeleteBarry
I wanted to say Thank You for providing this information, youre doing a great job with the site if you get a chance would you help me make some changes to my site? Ill pay you for your help, you can find the link to my site in my name. Thank you!
ReplyDeleteI thought Id share a quote with you from U.G. Krishnamurti that pertains to your blog, please dont take offense. You assume that there is such a thing as truth, you assume that there is such a thing as reality (ultimate or otherwise) – it is that assumption that is creating the problem, the suffering for you.That is just to make certain that your does not end up in the garbage can after a couple of days.
http://www.healthcaresups.com/spartagen-xt-exposed/
The certifications required for home health care aide agencies are not the same in different states. Ask them if they are certified by the state. Are they monitored by regulatory authorities? Make sure that they are licensed, insured and bonded so that everything is covered. Ask them about the kind of supervision procedures they follow.http://guidemesupplements.com/derma-breast-lift/
ReplyDelete