The reason macroeconomics doesn't work very well

I don't think it's politics (mostly). I don't think it's the culture of consensus and hierarchy. I don't think it's too much math or too little math. I don't think it's the misplaced assumptions of representative agents, flexible prices, efficient financial markets, rational expectations, etc.

Fundamentally, I think the problem is: Uninformative data.

I was planning to write a long post about this, and I never got around to it, and now Mark Thoma has written it better than I could have. So I'll just steal most of his excellent, awesome post, and add some boldface:

The blow-up over the Reinhart-Rogoff results reminds me of a point I’ve been meaning to make about our ability to use empirical methods to make progress in macroeconomics...it's about the quantity and quality of the data we use to draw important conclusions in macroeconomics. 

Everybody has been highly critical of theoretical macroeconomic models, DSGE models in particular, and for good reason. But the imaginative construction of theoretical models is not the biggest problem in macro – we can build reasonable models to explain just about anything. The biggest problem in macroeconomics is the inability of econometricians of all flavors (classical, Bayesian) to definitively choose one model over another, i.e. to sort between these imaginative constructions. We like to think or ourselves as scientists, but if data can’t settle our theoretical disputes – and it doesn’t appear that it can – then our claim for scientific validity has little or no merit. 

There are many reasons for this. For example, the use of historical rather than “all else equal” laboratory/experimental data makes it difficult to figure out if a particular relationship we find in the data reveals an important truth rather than a chance run that mimics a causal relationship. If we could do repeated experiments or compare data across countries (or other jurisdictions) without worrying about the “all else equal assumption” we’d could perhaps sort this out. It would be like repeated experiments. But, unfortunately, there are too many institutional differences and common shocks across countries to reliably treat each country as an independent, all else equal experiment. Without repeated experiments – with just one set of historical data for the US to rely upon – it is extraordinarily difficult to tell the difference between a spurious correlation and a true, noteworthy relationship in the data. 

Even so, if we had a very, very long time-series for a single country, and if certain regularity conditions persisted over time (e.g. no structural change), we might be able to answer important theoretical and policy questions (if the same policy is tried again and again over time within a country, we can sort out the random and the systematic effects). Unfortunately, the time period covered by a typical data set in macroeconomics is relatively short (so that very few useful policy experiments are contained in the available data, e.g. there are very few data points telling us how the economy reacts to fiscal policy in deep recessions). 

There is another problem with using historical as opposed to experimental data, testing theoretical models against data the researcher knows about when the model is built...It’s not really fair to test a theory against historical macroeconomic data, we all know what the data say and it would be foolish to build a model that is inconsistent with the historical data it was built to explain – of course the model will fit the data, who would be impressed by that? But a test against data that the investigator could not have known about when the theory was formulated is a different story – those tests are meaningful... 

By today, I thought, I would have almost double the data I had [in the 80s] and that would improve the precision of tests quite a bit... 

It didn’t work out that way. There was a big change in the Fed’s operating procedure in the early 1980s... 

So, here we are 25 years or so later and macroeconomists don’t have any more data at our disposal than we did when I was in graduate school. And if the structure of the economy keeps changing – as it will – the same will probably be true 25 years from now. We will either have to model the structural change explicitly (which isn’t easy, and attempts to model structural beaks often induce as much uncertainty as clarity), or continually discard historical data as time goes on (maybe big data, digital technology, theoretical advances, etc. will help?). 

The point is that for a variety of reasons – the lack of experimental data, small data sets, and important structural change foremost among them – empirical macroeconomics is not able to definitively say which competing model of the economy best explains the data. There are some questions we’ve been able to address successfully with empirical methods, e.g., there has been a big change in views about the effectiveness of monetary policy over the last few decades driven by empirical work. But for the most part empirical macro has not been able to settle important policy questions... 

I used to think that the accumulation of data along with ever improving empirical techniques would eventually allow us to answer important theoretical and policy questions. I haven’t completely lost faith, but it’s hard to be satisfied with our progress to date. It’s even more disappointing to see researchers overlooking these well-known, obvious problems – for example the lack pf precision and sensitivity to data errors that come with the reliance on just a few observations – to oversell their results. (emphasis mine)

This is the clearest and based statement of the problem that I've ever seen. (Update: More from Thoma here.)

I'd like to add one point about the limits of time-series econometrics. To do time-series, you really need two assumptions: 1) ergodicity, and 2) stationarity. Mark addressed the ergodicity problem when he talked about trend breaks. As for stationarity, it sometimes matters a lot - for example, if technology has a unit root, then positive technology shocks should cause recessions. But the statistical tests that we use to figure out if a time-series has a unit root or not all have very low power. There are some pretty deep theoretical reasons for this.

Anyway, that's just yet one more reason macro data is uninformative. That problem isn't going to be solved by gathering more accurate data, or by seeking out new macroeconomic aggregates to measure (though we should probably do both of those things anyway).

So what are the implications of this basic fundamental limitation of macro? I think there are three.

1. Beware of would-be prophets from outside the mainstream. There are a number of people, usually associated with alternative or "heterodox" schools of thought, who claim that macro's relative uselessness is based on an obviously faulty theoretical framework, and that all we have to do to get better macro is to use different kinds of theories - philosophical "praxeology", or chaotic systems of nonlinear ODEs, etc. I'm not saying those theories are wrong, but you should realize that they are all just alternative theories, not alternative empirics. The weakness of macro empirics means that we're going to be just as unable to pick between these funky alternatives as we are now unable to pick between various neoclassical DSGE models.

2. Macroeconomists should try to stop overselling their results. Just matching some of the moments of aggregate time series is way too low of a bar. When models are rejected by statistical tests (and I've heard it said that they all are!), that is important. When models have low out-of-sample forecasting power, that is important. These things should be noted and reported. Plausibility is not good enough. We need to fight against the urge to pretend we understand things that we don't understand.

3. To get better macro we need better micro. The fact that we haven't fond any "laws of macroeconomics" need not deter us; as many others have noted, with good understanding of the behavior of individual agents, we can simulate hypothetical macroeconomies and try to do economic "weather forecasting". We can also discard a whole slew of macro theories and models whose assumptions don't fit the facts of microeconomics. This itself is a very difficult project, but there are a lot of smart decision theorists, game theorists, and experimentalists working on this, so I'm hopeful that we can make some real progress there. (But again, beware of people saying "All we need to do is agent-based modeling." Without microfoundations we can believe in, any aggregation mechanism will just be garbage-in, garbage-out.)

After the financial crisis, a bunch of people realized how little macroeconomists do know. I think people are now slowly realizing just how little macroeconomists can know. There is a difference.