Thursday, September 29, 2011

Science is about discarding models, not making them

As someone who is preparing to apply for jobs as a professional economist, I can't help feeling down when I see Paul Krugman write something like this:
I’ve never liked the notion of talking about economic “science”...Still, when I was younger I firmly believed that economics was a field that progressed over time, that every generation knew more than the generation before. 
The question now is whether that’s still true...I think you can actually make the case that in important ways the profession knew more in 1971 than it does now... 
[M]any economists aren’t even trying to get at the truth. When I look at a lot of what prominent economists have been writing in response to the ongoing economic crisis, I see no sign of intellectual discomfort, no sense that a disaster their models made no allowance for is troubling them; I see only blithe invention of stories to rationalize the disaster... 
And all this makes me wonder what kind of an enterprise I’ve devoted my life to.
Paul Krugman is one of my personal heroes; it was the brilliance of his theories that lured me to the profession in the first place. So it makes me pretty sad to hear him questioning the value of the whole enterprise.

But I can't really disagree with his characterization. There's a reason you keep seeing these "is economics a science?" posts popping up around the blogosphere. The crisis made a lot of people question the value of economics, in its current form, as a useful tool for understanding and affecting the world. But when many economists seemed not to experience similar doubts, it raised the question of whether the profession approaches the world in a scientific way.

About 400 years ago, see, humans discovered a new way of understanding and interacting with the world, which seemed to work a lot better than what we had been doing before. The new approach involved things like doing experiments, writing down models, and observing the natural world. But what exactly was so great about this approach, which we called "science"? What were the key features of this new approach that made it so much more effective? If we have the models without the experiments, or vice versa, will we still get the same results?

I believe that we - some of us, anyway - have had a good answer to this question for a long time. The answer is that science works when it has a way to discard ideas that do not work. People have phrased this answer in different ways. Francis Bacon called it eliminative induction. Karl Popper called it "falsifiability." Richard Feynman called it "doubt." But it's the same idea. You actively look for what doesn't work, and throw it out.

Economists, I think, do not always see this as the essence of science. For example, here is Scott Sumner, in March 2011:
I’ve always thought that the debate over whether economics is a science is actually a debate about the meaning of the term ’science.’  If science is when people build models to better understand the world around us, then economics is a science.
I looked up a definition of "science," and came up with this:
The intellectual and practical activity encompassing the systematic study of the structure and behavior of the physical and natural world through observation and experiment. 
So, as economists we are systematic, we study the structure of the economy and the behavior of individuals in it, and we observe and experiment. Thus, apparently, we are a science.
Now, far be it from me to argue with the Oxford English Dictionary. But I contend that the real question here is substantive, not semantic. Whether or not an object is an "airplane," the important question is whether it flies. And whether or not economics is a "science," the important question is whether or not it discards its bad ideas. 

Economics has several ways of discarding bad ideas. The most common is to use empirical data to reject models. One particularly effective way of doing this is to use "natural experiments". A second method is to use experiments, which can be laboratory experiments or field experiments. 

So, procedurally, economics is a science.

But there is a difference between what a field can do and what it does do. A field is not just a set of techniques, it is also an established culture that chooses when and how to use those techniques. It is in this sense that economics does not always live up to its potential as a science. 

What I am not talking about are models that are that non-testable. Economics does make some of these, but that's fine. These models may be interesting as pure mathematics, or they may be of use in doing future science, even though they themselves are not science. I don't lose any sleep over these.

What bothers me is that the economics profession, as a culture, does not always insist that the testable models be tested and discarded. When enough economists ignore the facts and keep believing in models that can and should be discarded, then even though economics is procedurally a science, culturally it is not behaving like one. Obviously, I am referring to most of the popular macro models in use before the crisis.

So I disagree very strongly with Scott Sumner. The heart of science is not building models, but discarding them. Economics as a discipline has the tools to prune its tree of models, but economics as a culture has not been using these tools harshly enough. And so we've ended up as a sort of massive supermarket of models, where we have a model (or twenty models) for any conceivable phenomenon, but little way of knowing which model we should use in which situation. Economists who protest that "we had plenty of models that described financial crises" can't answer the question of why nobody was paying attention to those models before the crisis slapped us in the face.

This, I believe, is why the public has become incredibly skeptical of our profession. We let ourselves get away with too much. When we saw data that flatly contradicted our favorite theories, we just shrugged and kept on believing. And many of us seem still to be doing this, even now. And that is why these philosophy-of-science blog posts keep coming and coming. As well they should.

(Note: Most of what I'm complaining about here, like much of what I write on this blog, really goes only for macro. Micro, which is just a lot easier to test, is generally much better about being "scientific." But the difficulty of testing macro models doesn't let the field off the hook...)


  1. Anonymous7:09 PM

    i don't know... isn't part of the problem that macroeconomics discarded a bunch of fairly useful older models because they were not microfounded? and also economics in general discarded some relevant ideas and criticism from older philosophy-type schools of economic thought entirely because they didn't lend themselves to model building?

  2. As a first year PhD student who was also lured into economics by Krugman, I was hoping your post would cheer me up a bit. I'm not sure that it did.

  3. I've spent the last 3 or so years trying to disprove theories using econometrics and other methods. I think you're missing the broader issue, social sciences like economics deal with incredibly complex and dynamic systems which makes it incredibly hard to do thorough 'falsifications'. Even if you have some very sophisticated panel data multivariate regression models robust to stationarity etc.. finding significant effects contrary to a particular model, it is still normally seriously easy to poke holes in the test such as omitted variable bias (you'll probably be able to invalidate any regression model ever with this, it's always easy to find something the person didn't or simply cannot control for), problems with induction/extrapolation or endogeneity problems. This is why I tend to find many of the serious and top econometricians to be among the most skeptical of claims that any model, ever, has been empirically falsified. Basically there is normally always other explanations as to why a regression or 'lab test' (no external validity) produced results contrary to a theory that don't involve the theory actually being wrong.

    Bottom line is this, it might not have to do with an unwillingness to discard old models, but because social scientists simply don't have the data and tools at hand to be able to actually systematically disprove /most/ macroeconomic theories (however we're getting good at this in micro and other sub fields). I think that when people say that we're using models which have been thoroughly disproven, they're basically not very good at econometrics and overrate its power, or are simply lying.

  4. Anonymous10:22 PM

    From where I sit (which is not far from you geographically, actually), it looks hard to design and run experiments in macroeconomics. I gather response to shocks is central to testing models (cf. astrophysics, statistical mechanics). It's hard to make a lot of progress if you have to wait for a decade between experiments.

    (I guess what's getting PK down is that the abandonment of working models is the result of a long-term ideological struggle to poison your discipline. Good luck, fresh PhDs ... keep your wits sharp ... and your knives sharper!)

    There is one spectrum of research that spans hypothesis testing (i.e., evidence-based, like most medical research) to hypothesis generation (like a lot of engineering, where the central question is often "what the hell is going on here?"). Would you say macroeconomics is like engineering in that regard?

  5. Anonymous10:48 PM

    Well, once I got to grad school and found out what macroeconomists actually do, I switched to empirical micro.

  6. The basic point I'm hearing is that empirics is really, really hard when it comes to macro.

    Well, I completely agree.

    That's why I think macro got really bloated as a field pre-crisis, and is now taking its licks from an angry public.

  7. When enough economists ignore the facts and keep believing in models that can and should be discarded, then even though economics is procedurally a science, culturally it is not behaving like one.

    Carrying out something that looks like science, in exactly that manner, is Richard Feynman's definition of a cargo cult.

    You have joined a cult, Noah. Good luck.

  8. I do treat macroeconomics as a hard science , but the mainstream macro is a worthless science. It does not work well when it is most needed and economists completely deny their responsiblity for this idleness, what you call the absence of culture.
    I propose to start with data to build economics as a hard science ( It was the same way hundreds years ago - data (observations) drove Newton to formulate some mechanical links and to shape classical mechanics as a science.

    Macro was decoupled by force from the scientific mainstream in the 1930s when econometrics was introduced to connect economics and other hard sciences. When all models fail in terms of statistics (econometrics) of quantitative prediction of the past and the future the only way is to completely disregard scientific approach. Otherwise there is no ranking of scientists - all have failed.

  9. I think you need to decouple the philosophy of science from the professional ethics issue. Your real question is about the ethics of your profession. A throughly scientific profession could be wholly unethical (Eugenics, for example) and a thoroughly unscientific one could be ethical, and provide 'a useful tool for understanding and affecting the world' (Law might be an example).

    If you think about it, no-one keeps ideas that don't work. If someone keeps an idea that seems not to work, you haven't correctly understood the work that that idea is doing for them. In the specific case of economics, many practitioners are working to advance or defend certain political positions, rather than pursuing truth in a disinterested way.

    This means that the economic profession is not an altogether admirable one, but then we knew that. It doesn't of itself mean that one can't pursue an ethical and admirable career in that profession.

    To return to the analogy with law, most practitioners in the legal profession are engaged in something other than the disinterested pursuit of truth, and many are actively working towards unethical ends, but that doesn't make it impossible to practice law in an ethical way.

    What I think it does mean is that there has to be more than the law itself. If Krugman feels that he has devoted his life to the enterprise of economics, then I think he is right to have doubts because he has devoted himself to something which is less than admirable. Perhaps he needs to reconsider his goals and learn to see economics as a tool (or a series of tools) which he has used to pursue other things which are admirable in themselves.

    I would say it was obvious that economics provides a useful tool for understanding and affecting the world. Self-evidently people are using it for those purposes all the time. The question is to what ends?

  10. Anonymous7:59 AM

    Simply put economist became whores to corporate interest. If your source of income is tied to your master, your work becomes self-serving.

    Today you read more BS about how the models need tweaking; when any child can tell you the models were a failure.

    Sad to say but the level of corruption and cronyism is now so pervasive in this country it hard to see how we can find our way back to equilibrum.


  11. Anonymous9:05 AM

    As long as economists ignore the accounting that creates money, macroeconomics will be a pseudo-science.

    MMT is one of the few approaches to macro that remains true to the accounting. Steve Keen is close and has useful ideas on credit. The Wynne Godley models are excellent.

    We know that credit predates money - If you haven't read the stuff by David Graeber, you're uninformed - and should be the default starting position for thinking about macro.

    For example, S can't equal I, because it ignores the biggest player in the money economy, the government.

    Net financial assets for the private sector are created by the government if you use proper accounting.

    Some things are more true than others, and reading a bit of Neal Stephenson Anathem will help on this.

    Britonomist, if you use regression at all, its impossible to prove anything. Look up MMT and Wynne Godley and see what you find. The stocks/flows models of Godley are great for thinking through how money moves around the economy.

    Then the links between the real economy and the money economy become more clear too, when you use those models. Because there aren't leakages and you know there aren't leakages in the model of money, you can look to see how these sectors of the real economy behave in response.

    Models are about measurement at some basic level. If you can't measure something, you can't have a meaningful model of something. Because the natural rate of interest underlies many macro models, and we can't measure it, economics becomes an exercise in faith instead of a search for model errors.

  12. Anonymous9:43 AM

    "So I disagree very strongly with Scott Sumner. The heart of science is not building models, but discarding them. "
    Noah, I wish to introduce you the the concept of "truthification," proposed by Frank Wilczek to describe how he and David Gross worked out the asymptotic freedom of the gluon force. According to his account, they started out with a model which simply ignored a whole bunch of obviously important things. Sound familiar? Only in his case it lead to success. You can read the whole story (and much else) in "Lightness of Being."

    If you simply discard models that don't work you won't have anything to do when you get to work in the morning. Unacceptable. You need an alternative, not just a denial.

    More generally, I'm afraid your philosophy of science is out of date. I'm afraid this sounds simply snarky; I'm sorry about that, because I like your stuff and have learned a lot from it.

    If you want a more up-to-date account, try "Progress and Its Problems" by Larry Laudan. It's not new, but it's a lot more recent than what you've been reading. In the last couple of decades I haven't been keeping up; for the latest and greatest talk to someone in the Philosophy Department. Actually, don't; I want to see you finish.


  13. Anonymous12:20 PM

    Noah and Tom - rather than being disheartened, I would think you guys would be seeing this as a great opportunity. Don't young PhDs make their mark by overturning established ideas?

  14. @Jonathan:

    Yes, I know that one. The point is that Gross and Wilczek started with an unrealistic model, but they didn't end there. The goal was to get to something that worked.

    Thanks for the vote of confidence btw, I appreciate it. :)

    @Andrew & GeorgeK:

    Yes, politics definitely matters, and is one of the reasons macro is so dysfunctional. But the root of the problem is that macro is just insanely hard to get "right".


    Anathem is excellent. The Baroque Cycle is excellent as well.

  15. People have been making substantive criticisms of neoclassical models and their internal contradictions for decades and decades.

    Keynes was the first, although his contribution was pervereted and twisted into the neoclassical edifice - it now effectively consists of sticky prices/wages and IS/LM, and the only policy prescription is 'run a countercyclical fiscal policy'.

    Sraffra, Kaldor, Robinson, Minsky and others contributed to the effort, making substantial critiques of neoclassical models (particularly wrt the monetary & banking system).

    Recently, David Orrell has written an excellent book that probably criticises economics better than any other I've seen. Yves Smith, Steve Keen, Moshe Adler and Michael Hudson also have excellent critiques.

    How does economics deal with them? These people are cranks, ignore them. They don't understand economics fully, they don't understand epicycle we added to make our theory vaguely resemble reality. Forget the empirical evidence and the crisis - let's just carry on as normal!

    /rant over

  16. Anonymous9:04 PM

    I apolagize for my english. Unfortunately many economists are not popperians. I totally agree with you: economic science is about discard models when the theory doesn´t explain what is happening. In macro it is impossible to run experiments but we have the economic history to exclude bad ideas.

  17. If economics were a science the Cambridge capital controversy would have had an actual impact.

  18. @Britonomist - I agree with you about the complexity, but wonder whether you think that it is wise to pin our collective futures on models that are too complex to be understood, and hence too complex to be trusted?
    @Noah - great piece. Thanks

  19. Anonymous11:39 AM

    This looks like a conceptual question, 'Is economics a science'. It looks like its a matter of coming up with a characterization of the normal activities in the field of science, coming up with a definition of 'science' and seeing if the two match up. And in way that is what ought to be done. But remember that this question has a practical point. What professional economists would like to know is whether the activities they engage in are likely to get them at useful truths about human behavior. What lay people want to know is whether we have good reasons to defer to economists, in the way that we have good reason to defer to biologists, physicists, etc.

    So our definition of science ought to be such that if it implies that discipline X is a science, that we know the answers to those two questions. The definitions offered by Sumner and Williamson are not sufficient. To see why consider what else they let into the science club. The use of models to better understand the world? Well Hegel was doing that. He had a model (the dialectical model) which predicted all sorts of things about the world, like the number of planets, the perfect form of social system (luckily the one he was living in and working under), and all other manner of things. The reason Hegel wasn’t a scientist was because of how he developed his model, sitting alone in his armchair trying to deduce the fundamental structure of the world a priori from his concepts. This is the same thing that afflicted the earlier rationalists like Descartes and Leibniz. They were great mathematicians, but we don't remember then as great natural scientists, despite both of them spending a lot of time on natural science. The reason is that they thought the fundamental principles of their models could be reasoned out non-empirically. If the world seemed not to line up with what the model predicted, then the problem was with the interpretation we gave the data, not with the model.

    Williamson's definition would let in the Aristotlean corpus in its entirety. As the scientific revolution was a revolution against the Aristotlean approach to science, I think that is disqualifying by itself. It is not enough to be systematic and to observe the world (like others I am skeptical of the extent to which macro-economics can be thought of as experimental).

    The fundamental problem with thinking of systematic models as the hallmark of science is that it would make science a primarily deductive activity. You start with some assumptions that sound obvious to you (like Rational expectations) and then you just deduce consequences of those assumptions. I realize that is a caricature of what these people are doing, but the closer you get to that caricature, the farther you get from real science. Science is not mathematics and it is not philosophy. Those are primarily deductive activities. Science better not be like math and philosophy, since they don’t tell us how things actually behave and interact, and don’t pretend to.

  20. Well, I kept looking for what I had to say in the comments. Thankfully all I see are variations on a few themes. What I was thinking about is that there are economists who have had empirical falsification of their apparent models (basically that the liquidity trap really is real), but refuse to acknowledge it. And having failed to acknowledge it, they insist that their model is the correct one to which everyone should be paying attention. Unfortunately, the media acts like their model has not been disproven by recent events and they continue to get attention and funding for their research, etc. As for the whole external shocks being the basis of testing macroeconomics, that doesn't matter that much because there is a world of shocks out there.

    Anyone interested, I've got a blog where I discuss developments in the U.S. economy.

    Noah, I really liked this article you wrote.

  21. Thanks again for this great post Noah, and to all for the truly thought-provoking comments – I’ve quoted you all over here:
    In quoting you, I have also taken this general idea a couple of steps further in directions that I wonder whether you’ve considered?
    I admit that some of these directions might be a bit politically risqué for someone in your stage of (almost) professional life Noah, but would greatly appreciate any expert insights from you or your readers.
    Keep up the great work all!

  22. @Krugman - "I think you can actually make the case that in important ways the profession knew more in 1971 than it does now.."

    The same could be said about history, anthropology, archeology, sociology. In some fields the main emphasis should be on the transmission of old knowledge, not the discovery of new knowledge, a much more difficult task which, ironically, can take care of itself (genius will out).

    Outside the hard sciences we need to rethink the whole concept of research universities.

  23. I would like to respond to Britonomist.

    I am a scientist. All physical systems are extremely complex. We find ways to understand them by first looking at them simply and then gradually increasing the complexity of our analysis and subsequent models.

    Economics is not more complex, more difficult, harder to measure or less tractable than the physical universe.

    I agree with Noah. If you want to be considered a science, you'll need to adopt scientific methods. If you want to be creative and make lots of interesting models without critically evaluating them, then I suggest the new title Economic Architecture.

    Architects make all sorts of creative interesting models without considering if it is feasible to build that structure. They rely on a whole field of structural engineers to translate conceptual design into buildable design.

    In fact, now that I think about it - that may be the solution. A field of Economic Architecture for the model builders and a field of Economic Engineering for application of economics principles to policy. In which case, which well-known economists would you assign to each group?

  24. Anonymous8:30 AM

  25. The test of macro-economic models is macro-economic history, which is why we should study it more in relation to the economic ideas and policies that were influential at the time. Not much is going to be learned in any other way.