Wednesday, September 30, 2015

Theory vs. Data in economics

OK, I promised a more pompous/wanky followup to my last post about "theory vs. data", so here it is. What's really going on in econ? Here are my guesses.

First of all, there's a difference between empirics and empiricism. Empirics is just the practice of analyzing data. Empiricism is a philosophy - it's about how much you believe theories in the absence of data. You can be a pure theorist and still subscribe to empiricism - you just don't believe your theories (or anyone else's theories) until they've been successfully tested against data. Of course, empiricism isn't a binary, yes-or-no-thing, nor can it be quantitatively measured. It's just a general idea. Empiricism can encompass things like having diffuse priors, incorporating model uncertainty into decision-making, heavily penalizing Type 1 errors, etc.

Traditionally, econ doesn't seem to have been very empiricist. Economists had strong priors. They tended to believe their theories in the absence of evidence to the contrary - and since evidence of any kind was sparse before the IT Revolution, that meant that people believed a lot of untested theories. It was an age of great theoryderp.

That created a scientific culture that valued theory very highly. Valuable skills included the ability to make theories (math skill), and the ability to argue for your theories (rhetorical skill). Econ courses taught math skill, while econ seminars taught rhetorical skill.

Then came the IT Revolution, which dramatically reduced the costs of gathering data, transmitting data, and analyzing data. It became much much easier to do both high-quality empirical econ and low-quality empirical econ.

But at the same time, doing mediocre theory became easier and easier. The DSGE revolution established a paradigm - an example plus a framework - that made it really easy to do mediocre theory. Just make some assumptions, plug them into an RBC-type model, and see what pops out. With tools like Dynare, doing this kind of plug-and-chug theory became almost as easy as running regressions.

But Dynare and RBC didn't make it any easier to do really good theory. Really good theory requires either incorporating new math techniques, or coming up with new intuition. Computers still can't do that for us, and the supply of humans who can do that can't be easily increased.

So the supply of both good and mediocre empirics has increased, but only the supply of mediocre theory has increased. And demand for good papers - in the form of top-journal publications - is basically constant. The natural result is that empirical papers are crowding out theory papers.

But - and here comes some vigorous hand-waving - it takes some time for culture to adjust. Econ departments were slow to realize that these supply shifts would be as dramatic and swift as they were. So they focused too much on teaching people how to do (mediocre) theory, and not enough on teaching them how to do empirics. Plus you have all the old folks who learned to rely on theory in a theory-driven age. That probably left a lot of economists with skill mismatch, and those people are going to be mad.

At the same time (more hand-waving) the abruptness of the shift probably creates the fear that older economists - who review papers, grant tenure, etc. - won't be able to tell good empirical econ from mediocre. Hence, even empirical economists are quick to police the overuse of sloppy empirical methods, to separate the wheat from the chaff.

Now add two more factors - 1) philosophy, and 2) politics.

People have a deep-seated need to think we know how the world works. We have a very hard time living with uncertainty - most of us are not like Feynman. When all we have is theory, we believe it. We hate Popperianism - we recoil against the idea that we can only falsify theories, but never confirm them.

But when we have both facts and theory, and the two come into a local conflict, we tend to go with the facts over the theory. The stronger the facts (i.e. the more plausible the identification strategy seems), the more this is true.

The data revolution, especially the "credibility revolution" (natural experiments), means that more and more econ theories are getting locally falsified. But unlike in the lab sciences, where experiments allow you to test theories much more globally, these new facts are killing a lot of econ theories but not confirming many others. It's a Popperian nightmare. Local evidence is telling us a lot about what doesn't work, but not a lot about what does.

In physics it's easy to be a philosophical empiricist. As a physics theorist, you don't need to be afraid that the data will leave you adrift in the waters of existential uncertainty for very long. Physics is very non-Popperian - experimental evidence kills the bad theories, but it also confirms the good ones. In the early 20th century, a bunch of experimental results poked holes in classical theories, but quickly confirmed that relativity and quantum mechanics were good replacements. Crisis averted.

But that doesn't work in econ. A natural experiment can tell you that raising the minimum wage from $4.25 to $5.05 in New Jersey in 1992 didn't cause big drops in employment. But it doesn't tell you why. Since you can't easily repeat that natural experiment for other regions, other wage levels, and other time periods, you don't get a general understanding of how employment responds to minimum wages, or how labor markets work in general. Crisis not averted.

So philosophical empiricism is far more frightening for economists than for natural scientists. Living in a world of theoryderp is easy and comforting. Moving from that world into a Popperian void of uncertainty and frustration is a daunting prospect. But that is exactly what the credibility revolution demands.

So that's probably going to cause some instinctive pushback to the empirical revolution.

The final factor is politics. Theoretical priors tend to be influenced to some degree by politics (in sociology, that's usually left-wing politics, while in econ it tends to be more libertarian politics, though some left-wing politics is also out there). A long age of theoryderp created a certain mix of political opinions in the econ profession. New empirical results are certain to contradict those political biases in many cases. That's going to add to the pushback against empirics.

So there are a lot of reasons that the econ profession will tend to push back against the empirical tide: skill mismatch, the limitations of natural experiments, and the existing mix of political ideology.

Of course, all this is just my hand-waving guess as to what's going on in the profession. My guess is that econ will be dragged kicking and screaming into the empiricist fold, but will get there in the end.


  1. This seems like an oversimplification. It's not like there was no empiricism, and then IT came along and suddenly there were tons of it - econometrics existed before IT, even the Cowles commission stuff was long before IT. Certainly IT helped, but it's not true that economists weren't trained in econometrics prior to the IT revolution, not unless you go WAY back. Some of the most pioneering work in econometrics was done in the fifties and sixties.

    1. True, but it must have been really hard. When did economists first start analyzing natural experiments? Not until at least the 60s, right?

  2. Fairly reasonably pair of posts. I think part of the problem, which I don't think anybody has articulated so far, is that unlike in physics with the classical versus relativity plus quantum theory example, we do not have a nice neat competing broader theory sitting there. DSGE drew on the biggest theory around, general equlibirum, including its reliance on rational agents. Some of the most loudly proclaimed alternatives, such as Marxian theory, can be viewed from the abstract mathematical level as being just special cases of GE, although what really separates the Marxian theory off is its non-mathematical philosophical-historical-socilogical parts. But modern economists do not view that stuff as serious theory, which is mathematical one way or another.

    So, the long driip drip of local (and sometimes more than local) empirical findings that have weakened the theory has done just that, weakened the theory without obviously strengthening some neat alternative theory. If anything has been strengthened it has been behavioral economics, but that is notorious for not providing any nice general rules that can provide axioms for an alternative theory. We have a mess of things that seem to affect how people actually behave economically, but the strength of these various "anomolies" (some of which seem to be hardwired in neurologically) varies across people, and there are even people who fit the standard assumptions, have rational expectations, and all that (although they do not dominate markets as some theories have contended). We end up with a mess of a lot of modifications on the standard theory that do not readily parse themselves out into a new theory.

    We can see this playing itself out in macro modeling debate. So, we know that ratex is garbage empirically. General equilibrium as such may not be such a bad assumption, but when one extends it intertemporally (with ratex) one can quickly get nonsense. The DSGE game has in effect continued by adding lots of modifying bells and whistles as we find in the so-called "New Keynesian" versions, although a lot of these are not all that impressive, with the role of sticky prices in our macro fluctuations clearly way over-emphasized as indeed we have now quite a large lit that argues that we may see greater flucutations with flex prices, especially if they become tied to speculative bubbles, etc. The NK models stink, but such alternatives as ABMs, which aside maybe from network financial models have not really taken off, have not clearly claimed the day, and we have alternatives without any theory such as all the VAR models and their extensions, not to mention falling back on Old Keynesian models a la Krugman, possibly justified by handwaving about behavioral econ. I think macro modeling should use what we know about behavioral econ in modeling, but that is not a neat set of answers.

    A final point on this is indeed back on the problem of ratex. A major reason for the victory of ratex models in macro was that it gave a neat and pleasing answer to the problem of expectations formation: people expect what the "true model" forecasts on average, which was a neatly circular form of reasoning. If I think I have the correct model, then that is what people will expect (on average). This was much neater than parsing amongs all the competing formulations of adaptive (and other) expectations formations models. With the death of ratex, we have been thrown back into that swamp in reality, even as DSGE (and other) modelers resist admitting it.

    Barkley Rosser

  3. Anonymous3:48 PM

    I believe this clash between theory and empiricism only exposes the frailty of trying to do economics like physics or any natural science. While economists still sit on their imperial status and allow themselves to have a very poor understanding about the historical, sociological, political, geopolitical, antropological, etc, contexts of our objects, empirics will be only about destroying general theories and creating nothing worthy.

    General abstract theories in social sciences (economics included) should be regarded more as anectodes, fables, "ways of thinking" (something they are good at) than as actual "theories" in the strict sense (something they are very bad at). Focusing on CONTEXT (that is, history and actually existing economic, social and political structures) becomes much easier and less of an excuse for lack of technical rigour.

    So, we could be better by learning less micro and macro theory, and also be taught to treat them as reasoning techiques rather than scientific theories. On the other hand, it becomes ever more clearer that theoretical advances can only come with more (much more) studying of economic history and social and political structures of our objects. Data revolution both helps improving those analysis and screams for it.

  4. Scientists are supposed to be people who love solving puzzles, so they should be overjoyed with Popperianism: falsifying another theory means yet another puzzle to solve, with easy & less challenging solutions (i.e. boring solutions) already ruled out (Yeah!!!).

    If economists were true puzzle solvers they should be jumping up and down crying tears of pure unadulterated joy each and every time their theories get falsified by the hard cold facts generated by reality.

    If they don't, we should look at them askance... as mere politicians... the furthest thing from a truth loving puzzle solver that there is.

    I'm constantly asking economists "What would convince you that you're wrong about that?".... I'm surprised that I don't get many good answers. You'd think they'd be dying to tell you precisely what would falsify their theories, ... because the quicker a theory is shit canned, the sooner they can get back to the drawing board with a more challenging puzzle to savor.


    And if ultimately all economists come up with is that they don't deserve to be more than 0.01% confident about ANY theory, then so be it. If that's the truth, then they should love it dearly: and shit can their phony confidence levels as well.

    1. Anonymous7:32 PM

      I think most economists are thrilled by anomalies and unexpected results, and would agree that all their theories should be taken with a big lump of salt.

      The problem is that EVERYBODY wants economists to give them answers - Just about every time I tell a non-economist that I'm an economist they ask me what the government's economic policy should be, or whether there's a financial crisis coming, etc.

      What would physicists do if the government could adjust the amount of energy emitted by the sun? Even with infinitely better models, I'm sure the physics community would be divided about the optimal regulation of solar output. The doves would want to keep output high to stimulate plant growth. The hawks would worry about an overheating planet. You'd have Nobel Prize winning physicists with columns in the New York Times explaining solar policy and everyone would complain about how physics was too political. When someone decided to get a PhD in physics, their family members would tell them that they were ``glad someone's going to fix all the problems with our crop yields."

      Unfortunately, economics is in precisely this position. It's easy to be scientific when only the truth is at stake, but when there are a bunch of guys with models that say government debt is stifling the economy and we need austerity, and your models say austerity is stifling the economy and we need stimulus, and Congress and the President start asking economists what they should do, can you really tell them ``Yeah, we're about 0.01% confident that it's better to increase stimulus spending for XYZ reasons, but that's based on these absurd assumptions so it's totally possible that this advice is completely wrong."

      Sure, that's the scientific thing to do. But it's really hard for even the most ethical economist to stay in science-mode when the stakes are high and they're talking to non-experts. Even if all I want is what is best for the country, if I think my idea is even a little better in expectation than someone else's, it makes sense to talk mine up and publicly shit on theirs. It's not like I can explain to Congress why I think my method's probably a bit better than the other guy's in the present context. Why should I risk the possibility that the government will enact their policy instead of mine?

      That motivation is nothing compared to the prestige and financial rewards associated being the kind of economist who gives ``answers" professionally, whether it be as a government/corporate advisor, a pundit/columnist, or a popular blogger.

      We pay scientists to uncover the truth. We pay economists to give us their opinions - therein lies the difference.

    2. Yeah, except it doesn't work that way. On the one side, you have people saying "we should do some stimulus spending because we've seen it work over and over again". And on the other side you have these people saying "if you increase the supply of money, that's monetary inflation. And since monetary inflation and price inflation both contain the word 'inflation', that means that monetary inflation is price inflation."

    3. @Anonymous, thanks for your answer... I love it! I don't think I need to tell you I was being a bit tongue in cheek with my comment: It's slightly over the top... I've had that "speech" in my head for a while. I actually have seen physicists (Sean Carroll and Lawrence Krauss come to mind) say something amounting to a statement like this:

      "Well, I was hoping they *wouldn't* find the Higgs boson [or whatever], because that would have meant we still had another great puzzle to solve!!"

      And it occurred to me how different that is from the self assured and mutually far-from-consensus ideas that are out there in econ I see expressed by the corresponding media friendly economists on a regular basis. Of course Krauss and Carroll have lots of opinions on "fringe" ideas in physics too (how many universes ARE there?), but they *usually* (not always!) manage to insert a sizable helping of "this is speculation of course" qualifier into such musings.

      But I like your answer very much. Another thing I've thought of is that econ might be in the same place medical science was back in the 16th century: the king is sick, so SOMETHING has to me done... and so all the most notable physicians in the kingdom are summoned to examine him and give their advice... and immediately the blood letters start arguing with the ear candlers who argue with the cuppers and maybe even that guy in the corner who thinks maybe the mold on his bread might help (whom everyone else thinks is crazy!). Lol.

      But there you go, see? Medical science did go though something similar... a need to put it into practice prior to any verified consensus on the fundamentals. And yet they still have to deal with loud angry media savvy ignorant cranks... like the anti-vaxers and the "Food Babe" and her derp "army."

      And physics does get pressure to perform... in times of war (A-bomb, ICBMs, etc) perhaps sooner than they'd like. Perhaps in peace time as well (can we find an engineering solution to green house gasses? ... or for me, living in drought ridden California ... what ever happened to the idea of "seeding the sky" for rain?... unintended consequences not worked out yet perhaps? I guess I can find out.).

      @Cesium, I think my comment applies to you as well, if you were addressing me. Thanks.

  5. "the supply of humans who can do that can't be easily increased"
    Yes, the supply can be easily increased. It's a simple matter of universal education. Everyone in the world who wants to go to college, gets to go to college.

  6. A long age of theoryderp created a certain mix of political opinions in the econ profession. New empirical results are certain to contradict those political biases in many cases. That's going to add to the pushback against empirics.

    This is what I've long felt about economists - if the model doesn't comport with reality, then it's reality that has to be wrong.

    Ratex is not only nonsense, it's OBVIOUS nonsense to anyone who has ever dealt with real people in the real world. Everything that's based on it is an even higher level of nonsense.

    One root of the problem, IMHO, is that econ is inexorably connected to politics. This is that rare set of twins where both are evil.

    The other root is that the whole thing is too complex, with too many variables, and no understanding of what various states might actually mean. Real physics can't solve the 3 body problem.


    1. Question: What does it mean to "solve the 3 body problem"? See if you can answer without Googling... ;-)

    2. I will admit that I cannot. And it's too late - i already googled. The problem is chaotic. Moving one body alters its affect on the others. There are certain solutions for particular cases, but no general solution.

      The point is that the variables are not independent. In econ, this is also true. Further, the dependencies are likely non-linear, there are too many of them, and the actual number of them might not be known.

      So linear solutions that might valid approximations in a particular region of variable values will be lines tangent to curves. Move away from the region and the model becomes increasingly incorrect. Hence, frex, the Great Depression
      becomes the great vacation, unemployment insurance causes unemployment, etc.

      So even the best model will only be a special case.


    3. What would a "general solution" entail?

    4. The qualitative theory of nonlinear differential equations, which Poincare more or less invented while investigating and "solving" the three body problem in a 97 page paper.

  7. You write as if there is one kind of people. But there are at least two. Some people are just fine looking at evidence and changing theories to suit available evidence. Other people hold tight to mistaken beliefs regardless of evidence.

    Not only did we learn that raising the minimum wage in New Jersey didn't increase unemployment, but one side refused to believe that result and insisted that because their belief system said that increasing wages had to decrease employment, employment must have dropped. The other side found at least 11 distinct mechanisms that could account for an increase in employment in the face of rising minimum wages, and then went out and gathered evidence for those mechanisms.

  8. I'm not an academic, but watching from the outside, I haven't seen or heard much kicking and screaming. Empiricists just kinda quietly took over. But though I'm actually serious saying the only cure for empirics is more empirics, I'm not an optimist about the pace of likely progress. Empirics has not developed enough to take us out of the age of theoryderp nor will it anytime soon. Piketty and r > g being a clear illustration.

    1. "Piketty and r > g being a clear illustration." ?????

      Not sure what the reference is here. Why is that a clear illustration? (And what is it a clear illustration of.)

    2. Anonymous8:46 AM

      I suppose it is because the whole "r > g" polemic was a textbook case of economists focusing way too much on discussing a model and its specification rather than actual data and narrative argumentation.

    3. Piketty has on one hand done a lot of empirical work and on the other hand put forward one of the best examples of new theoryderp: the theory that as growth slows returns don't therefore wealth accumulates.

  9. The problem with this post is that it's all theory and no empirical work.

    1. :-)

      Was waiting for someone to say that...

  10. "In theory there's no difference between theory and practice, but in practice, there is"

  11. Much sounds reasonable. There is certainly and influx of data in certain areas of economics, but macro is one major exception. The primary goal is to explain the movement of variables like GDP that have quarterly frequency and persistence. Throw in potential structural breaks, and you have an econometric nightmare. There's not enough data to answer simple questions like 'Is there a unit root in GDP' with any confidence.

    There is some new data like the Billion Prices Project that might help, but empirical evidence in macro will be weak for the foreseeable future. Hence, macro theory will continue to get plenty of attention.

  12. GrueBleen7:54 PM

    If you are going to go on about how physics is empirically "better" than econ, then the very least you should do is look up Einstein's pronouncements about how even in physics (especially in physics ?) empirical evidence always underdetermines theory.

    Which is why Popperism doesn't actually work, even in physics. Oh yeah, fields of "knowledge" should at least be "theoretically" falsifiable, but just you try to do it when all your instruments and measures are deeply imbedded in your 'theory".

    What really happens is what Kuhn said: old theories just develop difficulties and problems until they get replaced by new (better ?) theories. Nothing is ever truly falsified. Nothing is ever confirmed.

    And perhaps you should also look up David Mermin's pronouncements on 'description' versus 'explanation', they are very relevant.

    1. This is why I said physics isn't actually Popperian! :-)

    2. GrueBleen7:42 PM

      Quite (but do look up Mermin if you're unfamiliar with him. Most entertaining.).

      However, physics "envy" isn't the main issue: the real question is whether Economics is Kuhnian (or maybe just Plankian).

  13. Noah,

    You aren't going back far enough in your economic history. Mathiness wasn't a feature of the early theorists - Adam Smith, Ricardo JS Mill etc. They described theory based on observation of the natural economic world around them. Mathematical notation then became a useful way to describe how a notional rational economic man would react to a set of circumstances (surrounded by a wide range of caveats - not the least of which being ceteris paribus). At some point the caveats fell away and people started believing that the maths wasn't a neat way to describe some interactions but was actually determining the behaviours.

    Economics at the moment is grappling with the fact that it keeps believing theory even after it falls at the first empirical test (see, for example, comparative advantage specialisation vs Leontieff Paradox). Even the Coase theorem is actually Coase's strawman explanation of property rights run wild without the body of his article which was about demonstrating that the institutional, legal and political framework, as well as informational asymmetry (or deficiency) would render a 'Coaseian' solution impracticable.

    1. scotra,

      Bad example with comparative advantage and Leontieff Paradox. That is only a paradox if one believes that there are only two factors of production: aggregate homogeneous labor and aggregate homogenous capital, as in various idiot textbook presentations. The minute one allows for heterogeneous labor, the "paradox" disappears empirically.

  14. The tyranny of popperism impedes the progression of knowledge and theory. The graviton may never be detected, but that doesn't render string theory useless or invalid.invalid. Theoretical physicists and economists have the tools to dismiss theories that are patently nonsense, so what remain are theories that are mathematically and logically consistent even if they cannot be adequately falsified.

  15. Though falsifiabililty is popularly associated with the name of Karl Popper it is a consequence of linking the model to logic via the meaning that is attached to the term "prediction." Thus, the phrase "tyranny of popperism" is equivalent to the phrase "tyranny of logic." The link to logic forged when a "prediction" is a kind of proposition.

    In the literature of global warming climatology, a "prediction" is a kind of proposition and is not a kind of proposition, a possibility because "prediction" is polysemic. When this term changes meaning in the midst of an argument this argument becomes an example of an "equivocation." Though an equivocation looks like a syllogism, it isn't one. Thus, though it is logically proper to draw a conclusion from a syllogism the same is not true of an equivocation. To draw such a conclusion is the "equivocation fallacy." This fallacy is of practical significance because drawing conclusions from equivocations is the sole basis for Obama administration regulatory policy on CO2 emissions.

    Under the circumstance that a "prediction" is a kind of proposition, it is possible for a climate model to provide a would be regulator of Earth's climate with information about the outcomes from his or her policy decisions in advance of these outcomes; thus, regulation is possible. Under the circumstance that a "prediction" is NOT a kind of proposition, it is impossible for a climate model to provide this information; thus regulation is impossible. Obama administration policy is based upon models of a type that provides no information. Regulation is impossible but is nonetheless being attempted by the Obama administration and at great cost to the economy of the U.S. Imposition of the "tyranny of logic" upon the Obama administration would head off this debacle.

    1. @Terry Oldberg:

      That's very interesting, but...when is a prediction not a proposition? Is that a well-known thing, in logic, rhetoric, etc.? Love to see links.


      No results found for "prediction is not a proposition".

    2. Steve Roth:

      Good question. Thanks for giving me the opportunity to clarify. A synopsis follows.

      When a "prediction" is a proposition and under the probabilistic logic this proposition has a probability of being true. The "probabilistic logic" is a generalization from the classical logic under which the truth values of propositions are replaced by probabilities of these propositions being true.
      Under the classical logic, information for a deductive conclusion from an argument is not missing. Under the probabilistic logic there is the possibility for a portion of this information (including all of it) to be missing.

      Science has a theoretical side and an empirical side. Probabilities lie on the theoretical side. The corresponding relative frequencies lie on the empirical side.

      A "relative frequency" is the ratio of two "frequencies." A frequency is a count of sampling units of a particular description in a sample drawn from the statistical population underlying the associated model.

      Eliminate the statistical population and there are no frequencies, relative frequencies or probabilities. There is no bothersome probabilistic logic or classical logic. This is the approach that has been taken by global warming climatologists in freeing their field of study from the tyranny of logic.

      One of the effects is to eliminate falsifability. For a global warming climatologist this has the beauty of enhancing job security.

      The article at provides details on the equivocation fallacy in global warming arguments plus citations to the scholarly literature on this topic.

    3. Terry,
      If global average termperature were to clearly start declining, some climate models would be found to have been falsified. Job security would only hold for those with tenure.

  16. Noah unrelated but there is a good MMT book/eBook here - understanding government finance:

    1. Anonymous11:15 PM

      Good MMT book is an oxymoron.


    "If global average temperature were to clearly start declining, some climate models would be found to have been falsified." Please provide a proof of your conjecture.

  18. Anonymous1:04 PM

    The question is not about empiricism, it's about the nature and quality of models to which data is applied.


  19. Anonymous2:00 AM

    Good use of the word "wanky", sir

  20. It's almost like the conceit of economics creating rationalizations for emergent effects of complex systems looks increasingly silly as data becomes increasingly more accessible to falsify such rationalizations. Or wait, maybe that's exactly what it's like.

    If you spent your life and earned great reputational acclaim for such a rationalization, would you be able to easily let it go in the face of evidence that you were basically an astrologer? It's a tough thing to ask of anyone.