Saturday, May 13, 2017

How should theory and evidence relate to each other?

In response to the empirical revolution in econ, and especially to the rise of quasi-experimental methods, a lot of older economists are naturally sticking up for theory. For example, here's Dan Hamermesh, reviewing Dani Rodrik's "Economics Rules":
Economics Rules notes with some approbation the rise of concern among applied economists, and especially labor economists, about causality. It fails, though, to observe that this newfound concentration has been accompanied, as Jeff Biddle and I show (History of Political Economy, forthcoming 2017), by diminished attention to model-building and to the use of models, which Rodrik rightly views as the centerpiece of economic research. He recognizes, however, that the “causation ├╝ber alles” approach (my term, not Rodrik’s) has made research in labor economics increasingly time- and place-specific. To a greater extent than in model-based research, our findings are likely to be less broadly applicable than those in the areas that Rodrik warns about. Implicit in his views is the notion that the work of labor and applied micro-economists might be more broadly relevant if the concern with causation were couched in economic modeling. If we thought a bit more about the “how” rather than paying attention solely to the “what,” the geographical and temporal applicability of our research might be enhanced... 
In the end, the basic idea of the book—that models are our stock in trade—is one that we need to pay more attention to in our research, our teaching, and our public professional personae. Without economic modeling, labor and other applied economists differ little from sociologists who are adept at using STATA.
Oooh, Hamermesh used the s-word! Harsh, man. Harsh.

Anyway, it's easy to dismiss rhetoric like this as old guys defending the value of their own human capital. If you came up in the 80s when an economist's main job was proving Propositions 1 and 2, and now all the kids want to do is diff-in-diff-in-diff, it's understandable that you could feel a bit displaced.

But Hamermesh does make one very good point here. Without a structural model, empirical results are only locally valid. And you don't really know how local "local" is. If you find that raising the minimum wage from $10 to $12 doesn't reduce employment much in Seattle, what does that really tell you about what would happen if you raised it from $10 to $15 in Baltimore?

That's a good reason to want a good structural model. With a good structural model, you can predict the effects of policies far away from the current state of the world.

In lots of sciences, it seems like that's exactly how structural models get used. If you want to predict how the climate will respond to an increase in CO2, you use a structural, microfounded climate model based on physics, not a simple linear model based on some quasi-experiment like a volcanic eruption. If you want to predict how fish populations will respond to an increase in pollutants, you use a structural, microfounded model based on ecology, biology, and chemistry, not a simple linear model based on some quasi-experiment like a past pollution episode.

That doesn't mean you don't do the quasi-experimental studies, of course. You do them in order to check to make sure your structural models are good. If the structural climate model gets a volcanic eruption wrong, you know you have to go back and reexamine the model. If the structural ecological model gets a pollution episode wrong, you know you have to rethink the model's assumptions. And so on.

If you want, you could call this approach "falsification", though really it's about finding good models as much as it's about killing bad ones.

Economics could, in principle, do the exact same thing. Suppose you want to predict the effects of labor policies like minimum wages, liberalization of migration, overtime rules, etc. You could make structural models, with things like search, general equilibrium, on-the-job learning, job ladders, consumption-leisure complementarities, wage bargaining, or whatever you like. Then you could check to make sure that the models agreed with the results of quasi-experimental studies - in other words, that they correctly predicted the results of minimum wage hikes, new overtime rules, or surges of immigration. Those structural models that failed to get the natural experiments wrong would be considered unfit for use, while those that got the natural experiments right would stay on the list of usable models. As time goes on, more and more natural experiments will shrink the set of usable models, while methodological innovations enlarges the set.

But in practice, I think what often happens in econ is more like the following:

1. Some papers make structural models, observe that these models can fit (or sort-of fit) a couple of stylized facts, and call it a day. Economists who like these theories (based on intuition, plausibility, or the fact that their dissertation adviser made the model) then use them for policy predictions forever after, without ever checking them rigorously against empirical evidence.

2. Other papers do purely empirical work, using simple linear models. Economists then use these linear models to make policy predictions ("Minimum wages don't have significant disemployment effects").

3. A third group of papers do empirical work, observe the results, and then make one structural model per paper to "explain" the empirical result they just found. These models are generally never used or seen again.

A lot of young, smart economists trying to make it in the academic world these days seem to write papers that fall into Group 3. This seems true in macro, at least, as Ricardo Reis shows in a recent essay. Reis worries that many of the theory sections that young smart economists are tacking on to the end of fundamentally empirical papers are actually pointless:
[I have a] decade-long frustration dealing with editors and journals that insist that one needs a model to look at data, which is only true in a redundant and meaningless way and leads to the dismissal of too many interesting statistics while wasting time on irrelevant theories.
It's easy to see this pro-forma model-making as a sort of conformity signaling - young, empirically-minded economists going the extra mile to prove that they don't think the work of the older "theory generation" (who are now their advisers, reviewers, editors and senior colleagues) was for naught.

But what is the result of all this pro-forma model-making? To some degree it's just a waste of time and effort, generating models that will never actually be used for anything. It might also contribute to the "chameleon" problem, by giving policy advisers an effectively infinite set of models to pick and choose from.

And most worryingly, it might block smart young empirically-minded economists from using structural models the way other scientists do - i.e., from trying to make models with consistently good out-of-sample predictive power. If model-making becomes a pro-forma exercise you do at the end of your empirical paper, models eventually become a joke. Ironically, old folks' insistence on constant use of theory could end up devaluing it.

Paul Romer worries about this in his "mathiness" essay:
[T]he new equilibrium: empirical work is science; theory is entertainment. Presenting a model is like doing a card trick. Everybody knows that there will be some sleight of hand. There is no intent to deceive because no one takes it seriously. 
In addition, there are also paper groups 1 and 2 to think about - the purely theoretical and purely empirical papers. There seems to be a disconnect between these two. Pure theory papers seem to rarely get checked against data, leading to an accumulation on the shelves of models that support any and every conclusion. Meanwhile, pure empirical papers don't often get used as guides to finding good structural models, but are simply linearly extrapolated.

In other words, econ seems too focused on "theory vs. evidence" instead of using the two in conjunction. And when they do get used in conjunction, it's often in a tacked-on, pro-forma sort of way, without a real meaningful interplay between the two. Of course, this is just my own limited experience, and there are whole fields - industrial organization, environmental economics, trade - that I have relatively limited contact with. So I could be over-generalizing. Nevertheless, I see very few economists explicitly calling for the kind of "combined approach" to modeling that exists in other sciences - i.e., using evidence to continuously restrict the set of usable models.


  1. eziblot5:53 PM

    The way out is the research programme. Instead of the random trickle of solo uncoordinated papers, list all the big Qs and puzzles and known unknowns about a given topic (along the lines you've followed for labour economics), hire a lot of researchers and build and test the models again and again until a reasonably complete knowledge is deemed to be acquired, do all the field research needed, involve anthropologists if needed, etc. Mini, focused Manhattan-projects. Sadly this requires government backing, not as much for the money but for the data access.

  2. Presumably some of of the problem is lack of specialists. Scientists are generally either theorists or experimentalists. So you get a healthy balance of both types of work and the most high impact papers are where they collaborate. Economists seem to more exclusively identify as theorists which probably isn't healthy.

  3. Hi, reading your post makes me think that the criteria used to assess the work of young economists seems overly restrictive and narrow.

    I suggest three broad categories for judging the merit of work of economists: (a) Is it useful?, (b) Is it elegant?, (c) Is it kind?

    The last category (Is it kind?) seems somewhat neglected. That said, many economists from Adam Smith to Tony Atkinson and Amartya Sen have turned their minds to this category, which I suggest concerns justice and fairness, and building societies that are resilient against climate change, pandemics, poverty, persecution (giving rise to refugees), prejudice in all forms (ethnicity, sex, religion), illiteracy & ignorance, corruption & extraction.

    1. This seem very misguided for something that wants to be a science. Shouldn't the first, and perhaps only, question be: is it true?

    2. Anonymous4:17 PM

      No. Given that economics is basically a social science and its recommendations are sometimes/often used to guide social policy, it cannot be free of value propositions (science can't be either). Even the basic questions of taxation take on an ethical/social dimensions when you ask "who is deserving of wealth?". Few conclusions in economics (if implemented) don't also carry often severe social consequences, so discussing economics without an ethical question (is it kind?) is kind of shortsighted.

  4. "In other words, econ seems too focused on "theory vs. evidence" instead of using the two in conjunction."

    This was something that I learned in research methods in Sociology - that a model has to be grounded in theory (and have both validity and reliability). There was no research methods course in Economics.

    Overall, I agree with you about 80%. The 20% disagreement is mainly a point of philosophy: is economics a social science, or a natural science; because how do you fit "culture" (which is a regional phenomena) and "work ethics" into labor models? That cannot be answered without philosophical questions of the validity of universality in the social sciences - something that Sociology also grapples with.

    What I truly appreciate about this article however, is the fact that it is truly self-reflective within the discipline. It asks very important questions about what Economics is versus what it wants to be, where it wants to go, and how to get to a new philosophical location. Those are things worth thinking about in order to make the study of the allocation of resources a relevant field of study in the future.

  5. Good good, your training is almost complete. Yes, feel the power of bayesianity.

  6. Is theory the siege tanks and empiricism the marines?? I like this blog post, and as a former Starcraft addict I love the photo at the top, but I'm having a hard time figuring out what they have to do with each other.

  7. "If you want to predict how fish populations will respond to an increase in pollutants, you use a structural, microfounded model based on ecology, biology, and chemistry, not a simple linear model based on some quasi-experiment like a past pollution episode."

    As someone who works in fish biology, I can say that this is just wrong. The response of fish populations to pollution depends on too many variables, many poorly known or unknown, for "microfounded" models to be very useful. Such models can tell you the consequences of how you think the world works, but will not tell you how it actually works.

    1. Anonymous9:49 AM

      Hi John,

      Do you have any links to some form of go-to model for a question like this or is it mainly ad-hoc computation?


  8. To the extent a model is written to permit empirical falsification, to the extent the norms of economics require such testability, and to the extent such tests get done and their results shape what's concluded, the model is scientific.

    To the extent that's not the case, models in economics more resemble religion with equations.

  9. Anonymous8:46 AM

    I'm a dumb sociologist but I know it's Stata, not STATA.

  10. This disconnect is obviously structural to the field. There is absolutely no incentive or need to make good models. That's what you get when a field is a self-referential talkathon. There is no penalty or consequences when models do no perform out of sample. Actually, there is no awareness for any such need.
    You guys get paid your little salaries no matter what, so why screw with the good thing?

    The empirical revolution will make things even worse. We are going to see billions of studies going in circles. The noise rider empire will live off of the public purse forever.

  11. In addition to the point Timothy Duignan makes about the experimental vs. theory specialization in the physical sciences, there is also a third category: phenomenology. It is usually non-trivial to translate and interesting theoretical speculation into a doable experiment. Phenomenologists work with both theorists and experimenters to figure out ways to test new theories.

  12. Every time someone says "I will use zzz as an instrument..." they are making theoretical claims. Imagine a linear model. A valid instrument should not be correlated with the error term. What is the error term. Many, perhaps most economic phenomena have multiple causes, several distinct paths leading to a particular outcome. The economist chooses one to estimate. The other causal pathways are lumped into the error term. So for an instrument to be valid, it should not be part of some other causal path. In fact, it should be a good instrument for a variable in some other causal path. You cannot successfully defend an instrument without discussing various causal explanations and either dismissing them or arguing that they are independent from your instrument. This discussion, rarely done, is theoretical in nature. I don't care one way or the other about equation manipulation, but I do care about implicit causal claims that are taken for granted and never defended.

  13. Anonymous4:28 AM

    You said "Oooh, Hamermesh used the s-word! Harsh, man. Harsh."
    I agree. A good illustration as to why it is harsh is given by Angus Deaton. In struggling to explain the findings that Case and he have found, he referred to a book by a sociologist about family life in a rust belt community. The combination of economic and social analysis get us closer to an answer in many cases. Alan H

  14. Smart young empirically-minded economists: another vain hope
    Comment on Noah Smith on ‘How should theory and evidence relate to each other?’

    Economics fits perfectly Feynman’s description of a cargo cult science: “They’re doing everything right. The form is perfect. ... But it doesn’t work. ... So I call these things cargo cult science, because they follow all the apparent precepts and forms of scientific investigation, but they’re missing something essential.”

    Economics have realized that science consists of two components theory and evidence: “Research is in fact a continuous discussion of the consistency of theories: formal consistency insofar as the discussion relates to the logical cohesion of what is asserted in joint theories; material consistency insofar as the agreement of observations with theories is concerned.” (Klant)

    But for some reason, economists do not get the two components properly together.#1 With regard to the criteria of material and formal consistency economics is a failed science.

    Currently, economists are in the mode of thorough self-critique. Yes, we have done too much math, yes, there has been too much abstract model building and too little empirics, etc. But then, this was what ‘thinking like an economist’ was meant to be: “It is a touchstone of accepted economics that all explanations must run in terms of the actions and reactions of individuals.” (Arrow)

    Economists readily admit obvious blunders. But then these are declared the faults of the old guard and now the smart young economists are on the right track. Fact is, though, that the new guard, just like the old, does not really understand what science is all about.

    Science is about general and invariant features of reality. In marked contrast, history is about unique event configurations on the surface which never repeat themselves. Science abstracts from all spatio-temporal details, which are so dear to the commonsenser and realist, because NO way leads from the history of falling apples to the universal Law of Falling Bodies. Nozick defines invariance in general terms: “An objective fact is one that is invariant under all admissible transformations.”

    Non-scientists and historians are glued to the ever changing surface, so they produce stories while scientists try to get hold of the underlying structure and produce laws=invariances. Economists are dimly aware of this: “That’s a good reason to want a good structural model.” (Noah Smith)

    The lethal defect of economics is that it is microfounded, i.e. based on behavioral axioms.#2 Now it holds that (i) there is NO such thing as an invariant of human behavior, and (ii), NO way leads from the explanation of human nature/behavior/action to the explanation of how the economic system works.

    Economics is NOT at all about human behavior, this is the subject matter of psychology, sociology, anthropology, biology, political science, etc., but about the behavior of the economic system. By consequence, economics has to be macrofounded.#3

    Economics has to step down as fake Queen of the so-called social sciences in order to eventually become a honorable member of the system sciences. This paradigm shift, clearly, is beyond the means of Noah Smith’s ‘smart young empirically-minded economists’.

    Egmont Kakarot-Handtke

    #1 See ‘What is so great about cargo cult science? or, How economists learned to stop worrying about failure’

    #2 See also ‘The happy end of the social science delusion’

    #3 See ‘True macrofoundations: the reset of economics’

  15. One class of models that combines structural theory and empricis is...(your favorite)...DSGE models. Of course, at present, they are of limited use for forecasting, but that's largely because the problem of building and fitting a structural macro model is so hard, so we still have other types of models for other tasks.

    Noah, instead of re-inventing the wheel every post, I'd recommend closely reading and translating Blanchard's

    for the masses.

    1. GW

      You say: “I’d recommend closely reading and translating Blanchard’s ”

      That is not such a good idea. Blanchard, too, is in deep methodological confusion. For details see:

      From the pluralism of false models to the true economic theory

      Eclecticism, anything goes, and the pluralism of false theories

      Getting out of IS-LM = Getting out of despair

      Egmont Kakarot-Handtke

  16. Noah, thanks for the great post. I agree combining theory and empiricism is the key. I think economics is all about a 2 step process:

    1. Make an economic statement (includes all models and theories)
    2. Determine validity of the statement

    I think it's important that economists take a step back and realize all they're doing is this 2 step process.