Thursday, March 17, 2016

Russ Roberts and the new empirical world

My last post, about Russ Roberts' EconTalk interview with David Autor, was requested by Russ as part of a general symposium on the topic. But it also reminded me that I had wanted to blog about two earlier EconTalk episodes - Russ' interviews with Joshua Angrist and Adam Ozimek.

These interviews, along with my own and Autor's, are basically part of a series. In this series, a bunch of people basically try to convince Russ that empirical economics matters a lot. Russ plays the part of the econ public - the average older academic, plus the econ-literate community of policy wonks, blog readers, and the like. He starts off basically thinking that econ is about theory (as it really was back when he was learning his chops). Then, through a successive series of conversations, he becomes convinced that not only have things changed, but that this is as it should be.

First, he talks to Angrist. Angrist is the chief academic evangelist for the "credibility revolution," the methodology at the heart of the new empirical revolution. Angrist tells him about a whole raft of exciting results that have come out of this methodological shift. Russ doesn't really know these methods very well, since only recently have they been codified, systematized, and popularized. So he doesn't quite know how to respond to Angrist's assertion that a revolution is underway. He keeps asking things like "But your new methods aren't convincing everybody...right?" And Angrist gets a little frustrated, basically responding that it's impossible to convince someone with evidence if they don't take a careful look at said evidence. If Russ were more up on this debate, he could have given Angrist a stiffer challenge, bringing up external validity concerns, or the fact that the new methods don't really work in macroeconomics. But remember, the whole point of this series of interviews is to introduce Russ into the new world! There will be time for pointed criticism later.

Second, Russ talks to me. I am a snarky, combative blogger who has trumpeted econ's empirical turn as something to be celebrated - something that will make the whole field more scientific. Russ and I have a great discussion and debate. Unlike Angrist, who focuses on research methodology and doesn't really bother going around asking people if they've changed their minds, I can offer a few concrete examples of economists who have shifted their views in response to empirical results, both of the "credibility revolution" type and other types like structural models and time-series macroeconometrics. I also rant about my favorite hobbyhorse, i.e. the value of empiricism. Russ is more convinced, but my combative nature probably leaves him uneasy with simply accepting my assertions.

Next, he talks to Adam. Adam makes basically the same points I do, but more eloquently and in a gentler fashion. Adam has been involved in an extensive debate with Russ over the value of convictions vs. evidence in economics. In his interview, Adam declares that although empirical results should be interpreted with humility, "stories and our own narratives about the world need even more humility," and we need more skepticism of our basic lenses [i.e. worldviews and ideologies] than we do of empiricism." Those are beautiful lines. Adam also cites empirical literatures that have convinced large numbers of economists to change their minds (to varying degrees) about big questions. Some are the same examples I cite, but he adds the example of trade, and cites Autor's work.

Russ is no longer nearly as dismissive of empirical econ, but he is still firmly in the role of the skeptic. He consistently refers to "sophisticated" econometric techniques, as opposed to "casual" evidence that he finds generally more convincing (he is not alone). The first few dozen times I heard Russ use the term "sophisticated" to refer to things like credibility-revolution econ, I was flummoxed, since many of these methods are simple comparisons of means. Suppose I tell my dog to bite your leg,* and I compare the number of bite marks after the attack to the number before - how sophisticated is that? But eventually I realize that by "sophisticated", Russ just means "systematic". He's saying that he trusts the evidence of his own eyes more than the opaque systematic data collection and analysis methods of empirical economists.

If I had realized that during my own interview, I would have pointed out that this is how science always works. Most people don't know how scientists know the distance from the Earth to Mars, or the mass of the electron, or that bacteria cause the plague. The scientific consensus on these topics was created by insiders - people with knowledge of both the theories and the empirical methods - convincing other insiders. Educated outsiders don't understand the theories and the methods, but they trust the scientists to get things right (and once in a while they even trust the scientists too much). Uneducated outsiders don't even trust the scientists - they believe vaccines cause autism, or that the world is flat, or that diseases are caused by demon possession, or whatever.

Russ isn't an uneducated outsider - he's an educated outsider in a field where the scientists (or whatever you want to call them, empirical researchers) haven't yet built up the trust and credibility to have their results gain instant acceptance by educated outsiders. Scientists cured the plague and put humans on the moon, but economists have a much harder job applying most of their results, since the results are usually about policy, and policy is ponderous, slow, clumsy, and subject to lots of random shocks. So Russ has not yet been shown the money, so to speak, and he's understandably wary.

But like most economists, Russ is smart, open-minded, and intellectually honest. In his interview with Adam, you can see that he is warming to the idea of the empirical revolution. Russ has been deeply disturbed by the ideological tone of the macroeconomic debates since 2008, and he recognizes that people might interpret casual evidence in ways that confirm their own worldview. He talks about psychological rigidity and people's tendency to defend their previous statements and positions. He worries that sloppy empirical work might be used by tendentious researchers as a front for their preferred policy positions (and this is a real worry), but he seems to realize that casual or anecdotal evidence is far more subject to this sort of misuse. Adam's contention that we need to be even more skeptical of our ideologies than of our evidence hits home.

So Russ is getting more into the empirical world here. In his interview with Autor, he doesn't question the validity of empiricism as a research method, but focuses on understanding the results in the context of theory. He does not ask many pointed questions about the research methodologies Autor uses, but he is not dismissive of the idea that empirical econ could poke holes in a hallowed policy consensus.

This is great to see. Though Russ is considerably more libertarian than the average economist, I think of EconTalk as a sort of a barometer for the wider econ world. The things Russ thinks out loud on the air must be things that thousands of older econ profs are thinking silently in their seminars, and that thousands of policy wonks are thinking silently while reading blog debates. The older generation came of age in a time where theory was king, and it's going to take some time to adjust to the new world. But that adjustment is happening.

So who should Russ have on EconTalk next? How should this series continue? I'd like to see some more top empirical researchers get a chance to strut their stuff. For example, he could get John Haltiwanger on to talk about declining dynamism in the American economy. He could get Raj Chetty to talk about mobility, education, and inequality. He could get Susan Athey to talk about machine learning techniques. Guests like this will continue to convince him that empirical econ is making lots of useful and reliable discoveries, as well as publicizing these discoveries to his many listeners!


Looks like I missed a couple of "episodes" in this "series". Here's Russ talking to Ed Leamer way back in 2010, and to James Heckman more recently.


  1. Jesse Shapiro and/or Matt Gentzkow would also be great guests for talking about the use of text analysis in economics

    1. This sounds interesting! Is it something like Franco Moretti's "Distant reading"? Could you post some links to their research?

    2. It's still quite new, but this:

  2. Interesting Noah. I've been following an interesting debate on empiricism in economics between two amateurs: John Handley and Jason Smith. Here's one of John's posts that will lead into the discussion.

    As you can see by John's tag line (and his posts), he's someone to keep your eye on for the future.

    1. ...and here I ask another amateur (PKE & MMT advocate) Brian Romanchuk "How would you know if you're wrong?" (my favorite question to economists: amateurs or otherwise).

    2. IMO this guy does a decent job of showing (in six short videos) how the theory of evolution meets the six criteria proposed by the governing board of the Council of National Science Education Standards for defining a science:

      1. Observational data
      2. Accurate predictions
      3. Logical
      4. Open to criticism
      5. Accurate information
      6. No presupposition

      How would economics stack up? What would the videos look like?

    3. Tom, I ask this genuinely: Do you really find those blog posts stimulating? They strike me as the kiddie pool of this sort of discourse. "You can't control for everything" is the same kind of tired nihilist nonsense that Russ spouts, and it ignores decades of work critically thinking about how you can, well, control for everything on some occasions. Especially on the minimum wage, a topic which was the breeding ground for this work.

    4. Frog, in this dispute between Jason and John I favor Jason's view, which argues that empirical evidence IS sufficient to eliminate models provided the models have a level of complexity which is commiserate with the data available (and models who's complexity exceeds this threshold have no business being that complex to begin with at this state of the game).

      However, I don't favor that view from any personal experience, but rather from holding to what I take to be an optimistic attitude on the matter. I think John's view is pessimistic, and he hasn't convinced me that his pessimism is warranted. However, I'll be the first to admit that maybe it IS warranted. I'm certainly not the one to be able to judge.

    5. Frog, John has another post up on the subject. I don't know that I agree or even understand everything he's saying. I don't think Jason, for example, would describe his view as exactly the same as Poppers: there's some daylight there.

      Interestingly, physicist Sean Carroll has nominated "falsifiability" as an idea that's "ready for retirement." I found that curious, but here's his argument.

  3. Anonymous2:31 PM

    I was the snarky commenter in the prior blog post. Three more data points to add to your list of his podcasts, which I listen to selectively depending on my interests and his guests. You can find these easily, and none involves an econometrician discussing regression results.

    1. Interview with Christopher Hitchens, nominally on George Orwell. Roberts' basic take was that Orwell was an anticommunist libertarian (like Roberts). Hitchens talked patiently about Owell's evolution (from anti-colonialism to anti-racism to anti-totalitarianism) and his untimely death due to poverty. No impact. Orwell was a libertarian.

    2. Interview with Anat Admati, nominally on banking capital. Roberts' basic take was, despite the financial crisis, that finance was jim dandy. Admati patiently walked Roberts through a typical bank's balance sheet, explaining that most banks fund their lending operations through debt and leverage, unlike most industries. She also exposed the lie at the heart of bankers' argument against reserve capital (what we hold back, we can't lend). No impact. Finance is jim dandy.

    3. Interview John Cochrane's father in law, what's-his-name, nominally on the EMH. Roberts' basic take was that the EMH is an eternal verity. Fama talked vaguely about the joint hypothesis of EMH that could never let it be rejected. No discussion of strong form versus weak form. No criticism. Just glee in the eternal verity.

    Roberts goes into every interview with his strong libertarian prior, and leaves with an identical posterior. No amount of data will sway him. Much like your BFF, John Cochrane. To paraphrase DeLong, Republican economists are very much Republican first and foremost, City Smiths and Morgan Men through and through.

    1. Anonymous5:51 PM

      I wouldn't want anyone else to host EconTalk, but it's definitely striking to notice how differently Russ behaves with people that totally confirm his bias (Ed Leamer, John Taylor, Gene Fama) and people that don't (Dick Thaler, Joe Stiglitz, Josh Angrist).

  4. Anonymous2:44 PM

    However convinced he's becoming about the successes of micro empirics, he's probably still firmly skeptical about macro/finance empirics. Two representative episodes along these lines are with Lars Hansen in 2014 and Valerie Ramey in 2011.

    In the Hansen episode, Russ says at one point "It's possible that if you are in a financial setting where you have to take risks, you are better off not quantifying them because that fools you into thinking that it's safer than it is." It's one of the most striking things I've heard him say.

    The Ramey episode is about estimating government spending multipliers, which is a topic where Russ seems to have an ideological axe to grind. At the end of the episode, Russ asks Ramey to discuss the work of (then) recently announced Nobelists Sargent and Sims, and he makes a beeline for a question about the problems with VAR methods.

    I'd love for Chris Sims to appear on EconTalk. The Bayesian approach to empirical research has been pretty underrepresented on the show, and I think Chris Sims would be an especially pointed and articulate representative of the kind of empirical work that Sargent, Hansen, and others do.

    1. Anonymous3:01 PM

      + 1. Sims would tear Roberts a new one.

      Folks would be better served listening to Barry Ritholtz's podcast on Bloomberg. The financial wizards and plutocrats all acknowledge the vast role of luck in their lives. Well, all but one; Cliff Asness, who's the smartest guy in the room unless John Cochrane or his father in law are present.

  5. Not sure if it's part of this symposium, but Arindrajit Dube.

  6. Anonymous11:48 PM

    "If Russ were more up on this debate, he could have given Angrist a stiffer challenge, bringing up external validity concerns, or the fact that the new methods don't really work in macroeconomics."

    I agree with both of these, but there's an equally important issue: false positives due to p-hacking, publication bias, the garden of forking paths etc. (see Gelman:

    Empirical social psychology is facing a crisis (a nice overview is here: And the crisis is in research based on randomised control trials, where endogeneity is no concern! I'll grant that sample sizes are generally a lot larger in empirical economics, so we may not face extremely low power as often. But there is still so much room for dropping observations, adding and removing controls, running regressions on different dependent variables, and running regressions on different sub-samples that it's extremely easy to present noise as truth.

    Angrist and Pische's credibility revolution is only the beginning of what's needed. We don't yet know how bad the reproducibility crisis is in economics.

  7. "Russ is smart, open-minded, and intellectually honest"

    It's nice of you to say this, but, ironically, not really backed up by much empirical evidence.

    It's bizarre to call Russ's anti-empirical mindset the one of the average older academic. Orley Ashenfelter is 73. Ed Leamer is 71. Robert LaLonde is probably roughly the same age. You have to get extremely crusty before you start hitting economists who were completely unaware of what was going on in the late 80s. Even the fogiest fogie theorists at top departments (including my own) don't have the nihilist views of Russ. It's not just anti-empirical, it's anti-intellectual.

  8. Even after the Bloomberg piece, I don't understand the debt analogy, which seems to me to miss the role of the medium of exchange.

    To make a different analogy, if Noah uses his labor to write an economics paper and agrees to share it with me in exchange for a promise to provide X hours of legal work in the future, I've got a "trade deficit" and a debt to Noah that will some day become due. Same if the agreement is a mere promise to pay.

    But in economy with multiple actors and a medium of exchange, I exchange my currency for his economics paper and have no debt to him, even by analogy, even though I've got a "trade deficit." He's received his value in currency which he can exchange with other actors for real value. And that's still true even if upon receiving USD from me, Noah goes out and exchanges them for Bitcoin, his preferred form of money.

    Which seems like what typically happens with trade. Leaving aside tax and other legal consequences that keep money from moving, a German car maker that sells me a car in the US for USD is going to go out and sell those dollars for Euro and that will be the end of any "debt" resulting from the transaction. There's been a wealth transfer from the US to Germany, and that may not be indefinitely sustainable, but I don't see why there is necessarily a future transfer of "something of real value" in the other direction in the future. The German company (or German citizens as a whole) could take use that wealth to purchase something from the US in the future, but they could also purchase something from China instead.

    Or, in other words, what is the mechanism through which Germany not only will but must extract future "real value" from the US to offset my car purchase?