What can you do with a DSGE model?

When the Bank of England invited me to give a talk at their workshop on macroeconomics, I wasn't sure if they wanted me to provoke (i.e. troll) them with the kind of skeptical stuff I usually write on this blog, or to talk about my own research on artificial markets and expectations. So I did both. Now, this is a central bank event, which means secrecy prevails - so I can't tell you what the reaction was to my talk, or what other people said in theirs. But I thought I'd reproduce part of my talk in a blog post - the part where I talked about DSGE models. (In other words, the provocative part.)

"DSGE" is a loose term. It usually implies much more than dynamics, stochastics, and general equilibrium; colloquially, to be "DSGE" your model probably has to have things like infinitely far-sighted rational expectations, rapid clearing of goods markets, certain simple types of agent aggregation, etc. So when I talk about "DSGE models", I'm loosely referring to ones whose form is based on the 1982 Kydland & Prescott "RBC" model.

In recent times, of course, RBC models themselves have fallen out of favor somewhat in the mainstream business-cycle-modeling community, and have gone on to colonize other fields like asset pricing, international finance, and labor econ. As of 2013, the most "mainstream" DSGE models of the business cycle are "New Keynesian" models. The most important of these is the Smets-Wouters model, which has gained a huge amount of attention, especially from central banks, for seeming to be able to forecast the macroeconomy better than certain popular alternative approaches. If you know only one DSGE model, Smets-Wouters is the one you should know.

Anyway, my talk asked the question: "What can you do with a DSGE model?" Most people who evaluate the DSGE paradigm don't focus on this question; they either trace the historical reasons for the adoption of DSGE (the Lucas Critique, etc.), or they discuss the ways DSGE models might be improved. Instead, in my talk, I wanted to take the perspective of an alien econ prof who showed up on Earth in 2013 and tried to evaluate what human macroeconomic theorists were doing.

A DSGE model is just a tool. It's a gizmo, like a fork lift or a lithium-ion battery. The U.S. and Europe have invested an enormous amount of intellectual capital - thousands of person-years of our best and brightest minds - in creating, testing, and using these tools.

So what can you do with these tools?


1. Forecast the economy?

One thing you might want to do with a business cycle model is to forecast the business cycle. DSGE models have improved enormously in this regard. Though early RBC models were notoriously bad at forecasting, more recent, complex DSGE models have proven much better, and are now considered slightly better than vector autoregressions, and about as good as the Fed's own forecasts.

But as Rochelle Edge and Refet Gurkaynak show in their seminal 2010 paper, even the best DSGE models have very low forecasting power. Check out these tables from that paper:

These tables show the forecasting performance for the Smets-Wouters model (which, remember, is the "best in class") from 1992 through 2006. The first table is for inflation forecasts, the second is for growth forecasts. Look at the R-squared values. These numbers loosely describe the amount of the actual macroeconomic aggregate (inflation or growth) that the model was able to predict. An R-squared of 1 would mean that the forecasts were perfect. You'll notice that most of the numbers are very, very low. The Smets-Wouters model was able to predict a bit of inflation one quarter out (though the Fed's internal forecasts were much better at that horizon), and not at all after one quarter. As for growth, the DSGE model had very low forecasting power even one quarter ahead.

Now, this doesn't necessarily mean that DSGE models are sub-optimal forecasters. These things might just be very very hard to predict! Humanity may simply not have any good tools (yet) for predicting macroeconomies, just like we aren't yet able to predict earthquakes.

But there's also some evidence that we could be doing better than we are. In this 2013 paper, Gurkaynak et al. test the "forecast efficiency" of DSGE models, and find that their forecasts are not optimal forecasts. Also, they find that simple univariate AR models are often significantly better at forecasting things like inflation and GDP growth than the best available DSGE models! This is not an encouraging finding for the DSGE paradigm, since AR models are just about the simplest thing you can use.

Also, in this discussion of forecasting, remember that the deck has already been stacked in favor of DSGE models. Why? Because of publicity bias and overfitting. If DSGE models don't do well at forecasting, researchers will add features until they do better. As soon as they do well enough to look good, researchers will publicize the success. This is a perfectly appropriate thing to do, of course - it's like improving any machine until it's good enough to sell. But it means that the publicized models will have a tendency to overfit the data, meaning that their out-of-sample performance will usually be worse than their in-sample and pseudo-out-of-sample performance.

(Update: Via a commenter, here's a good survey of DSGE models' forecasting ability, including how they did in the Great Recession. See my new post for more...)

In other words, DSGE models are probably not very good as forecasting tools...yet. But they're about as good as anything else we have. And they have improved considerably compared to their early incarnations.


2. Give policy advice?

This is what DSGE models are "supposed to do" - in other words, most academics will tell you that this is the purpose of the models. Actually, a model can be perfectly good for policy advice even if it's bad at forecasting. This is because forecasts have to deal with lots of different effects and noise and stuff that's all happening simultaneously, while policy advice only requires you to understand one phenomenon in isolation.

But here's the problem: To get good policy advice, you need to know which model to use, and when. So how do you choose between the various DSGE models? After all, there's a million and one of them out there. And they're usually mutually contradictory; since they're fitted using many of the same macroeconomic time-series (e.g. U.S. post-WW2 GDP, employment, and inflation), one of them being a good model (even just in one specific situation) means the others must then not be good models.

So how do you choose which model to use to give you advice? Old methods like "moment matching", which were used to "validate" the original RBC models, are, simply put, not very helpful at all.

What about hypothesis testing? Again, not very helpful. If you make the model itself the null, then of course you'll reject it, because any model will be too simplified to explain everything that's going on in the economy. If you make the null the hypothesis that the DSGE model parameters equal zero, you'll almost always reject that null, even if the model is grossly misspecified.

In principle, I think you should use some kind of goodness-of-fit criterion, like an R-squared, using out-of-sample data and adjusted to favor parsimonious models. At the macro conferences and seminars I've attended, I haven't see people saying "Look at the out-of-sample adjusted R-squared of this model! We should use this one for policy!" Maybe they do say this, though, and I just haven't seen it. (Update: Here, some people, including Smets and Wouters, do evaluate the fit! Definitely check out this paper if you're into macro modeling.)

But anyway, there's a few more problems here. One is the lack of clearly defined scope conditions; macro theorists rarely work on the difficult problem of when to stop using one model and start using another (see next section). Another is the nonlinearity problem; most DSGE models are linearized, which makes them easier (i.e. possible) to work with, but means that their policy recommendations often don't even match the model.

(As an aside, many people say "OK, we don't know which DSGE model is right, so just combine a bunch of models, with some weights." Fine...but the weights aren't structural parameters, so by doing this you give up the supposed "structural-ness" of DSGE models, which is the main reason people use DSGE models instead of a spreadsheet in the first place.)

So to sum up, DSGE models could offer policy advice if you used an appropriate model selection criterion, and dealt carefully with a bunch of other thorny issues, AND happened to find a model that seemed to fit the data decently well under some clearly defined set of observable conditions. But I don't think we seem to be there yet.


3. Map from DSGE models to policy advice?

OK, so it's really hard to give definitive policy advice with DSGE models. Maybe you could instead use DSGE models as maps from policymakers' assumptions to policy advice? I.e., you could say "Hey, policymaker, if you believe A and B and C, then here are the implications for policies X and Y and Z." In other words, since DSGE models are internally consistent, maybe they can help tell policymakers what they themselves think can be done with regards to the macroeconomy. (Another way of saying this is that maybe we can leave model selection up to the priors of the policymaker.)

There's just one problem with this. DSGE models are highly stylized, meaning that it's often not possible even to figure out whether you buy an assumption or not.

Let me demonstrate this. Let's take a look at a DSGE model - say, Christiano, Eichenbaum, and Evans (2005). This New Keynesian model is very similar to the Smets-Wouters model mentioned above. Here is a VERY truncated list of the assumptions necessary for this model to work:

• Production consists of many intermediate goods, produced by monopolists, and one single consumption good" that is a CES combination of all the intermediate goods.
• Firms who produce the consumption good make no profits.
• Firms rent their capital in a perfectly competitive market.
• Firms hire labor in a perfectly competitive market.
• New firms cannot enter into, or exit from, markets.
• All capital is owned by households, and firms act to maximize profits (no agency problems).
• Firms can only change their prices at random times. These times are all independent of each other, and independent of anything about the firm, and independent of anything in the wider economy. (This is "Calvo pricing". The magic entity that allows some firms to change their prices is called the "Calvo Fairy").
• The wage demanded by households is also subject to Calvo pricing (i.e. it can only be changed at random times).
• Households purchase financial securities whose payoffs depend on whether the household is able to reoptimize its wage decision or not. Because they purchase these odd financial assets, all households have the same amount of of consumption and asset holdings.
• Households derive utility from the change in their consumption, not from its level ("habit formation"). Households also don't like to work.
• Households are rational, forward-looking, and utility-maximizing.

OK, I'll stop. Like I said, this is a VERY truncated list; the full list is maybe two or three times this long.

How many of these assumptions do you believe? I'm not sure that's even possible to answer. Formally, most of these are false. Some are very obviously false. The question is how good an approximation of reality they are. But how do we know that either?? Is it a good approximation of reality to say that households purchase financial securities whose payoffs depend on whether the household is able to reoptimize its wage decision or not? How would I even know? 

In principle, you could look at the micro evidence and see which of these assumptions looks kinda-sorta like real micro behavior. Some people have tried to do that with a few of the assumptions of the Smets-Wouters model; their results are not exactly encouraging. But if you tried to go ask a policymaker "Which of these things do you believe?", you'd get a blank stare.

So DSGE models don't make a clear map from assumptions to conclusions. But how about using them just to explore the robustness of models to variations in assumptions? A central bank (or the academic macro community) could make a bunch of DSGE models and compare their results, just to see how different modeling assumptions affect conclusions. In fact, that's probably what the academic macro community has been doing for the past 30 years. This seems somewhat useful to me, but there's a problem. DSGE models are not very tractable, so it's probably the case that nearly all of the modeling assumptions usable in DSGE models are poor approximations of reality. In that case, we'll be stuck searching next to the lamppost.

4. Communicate ideas?

DSGE models can definitely be used as a language in which to communicate ideas about how the economy works. But they are probably not the best such language. Simpler econ models, like OLG models, or even partial-equilibrium models, are much more flexible, and can be understood much more quickly by an interlocutor. DSGE models have a ton of moving parts, and it's generally very hard to see which assumptions end up causing which results. The better a model matches data or forecasts future data, the more moving parts it will generally have. This is called the "realism-tractability tradeoff". 

So if you only work with DSGE models, and if you try to understand everything in terms of DSGE models, you'll have a hard time communicating with other economists. I can see this being a problem in a central bank, where people need to communicate ideas very quickly in times of crisis.

So, what else would you have us do?

There are a number of alternatives that have been proposed to DSGE models. Different alternatives are generally proposed for the different purposes listed above.

For communicating ideas, the most popular alternatives are simpler, OLG-type models (which are, technically, DSGE, though not what we typically call "DSGE"!), and partial-equilibrium models (suggested by Robert Solow). I've seen some people use these at seminars, especially the OLG type, so I think this alternative may be catching on.

For forecasting, the common alternatives are "spreadsheet" type models (Chris Sims' dismissive term) that don't assume structural-ness. This is the kind of model used by the Fed (the FRB/US) and by some private forecasting firms like Macroadvisers.

Policy advice is the thorniest question, since you need your model to be structural. For this, the main alternative that has been put forth is called "agent-based modeling". I don't know too much about this, and the name is weird, because DSGE models are also agent-based. But basically what it seems to mean is to specify a set of microfoundations (behavioral rules for agents), and then do a big simulation. The big difference between this and DSGE is that with DSGE you can write down a set of equations that supposedly govern the macroeconomy, and with ABM you can't.

So are we wasting our time making all these DSGE models, or not?

My answer is: I'm not sure. So far, we don't seem to have gotten a heck of a lot of a return from the massive amount of intellectual capital that we have invested in making, exploring, and applying these models. In principle, though, there's no reason why they can't be useful. They have flaws, but not any clear "fatal flaw". They're not the only game in town, and realization of that fact seems to be slowly spreading, though cultural momentum may mean that the more recently invented alternatives (ABM) will take decades to catch up in popularity, if they ever do.