Wednesday, January 08, 2014
Bad event studies
Author: Noah Smith
Blogs are great, but they have their limitations. A lot of macro bloggers want to use specific real-world events to validate or invalidate their theories - "Market Monetarism", "Keynesianism", and so forth. This is not bad if done in a sort of Bayesian way - in other words, looking at events and slightly modifying or updating one's worldview. For example, the events since 2007 have gradually made people less and less worried about inflation. But when done in a small-sample-Frequentist sort of way - looking at a single event and viewing it as an up-or-down test of a grand theory - this kind of inference is incredibly inaccurate.
For example, take the recent row over the year 2013, in which we saw the Sequester and QE3. Did the events of 2013 disprove Market Monetarism? Paul Krugman at one point said the year would be a test of that (fuzzily defined) theory. Mike Konczal said much the same thing. Now Scott Sumner and David Beckworth are saying (surprise surprise) that 2013 demonstrates considerable empirical support for Market Monetarism. Sumner also lays into me for tweeting Konczal's article (though when I tweet things with question marks, it means I'm skeptical of the thing I'm tweeting; Sumner probably doesn't know that).
Brad DeLong points out something very important, which is that 2013 was a real experiment only if we believe there was a regime change in policy. In fact, there may have been zero, one, or two regime changes - the Sequester could have been a big departure, or a continuation of what people expected to happen, and the same goes for QE3. If 2013 was a regime change in monetary but not fiscal policy, then our conclusions should be different than if 2013 was a regime change in fiscal but not monetary policy, and yet different if 2013 was a regime change in both fiscal and monetary policy, and still different if 2013 was a regime change in neither fiscal nor monetary policy. And it is hard to know what represents a real regime change.
So that's one problem with doing these kind of informal "event studies" on blogs.
A second problem is that the theories themselves, as presented on the blogs, are fuzzy and not well-defined. What does "Market Monetarism" say? Does it say that fiscal policy doesn't affect the real economy, and that only monetary policy has an effect? Or does it say that monetary policy and fiscal policy both have an effect? Does "Keynesianism" say QE works at the Zero Lower Bound, or not?
But the biggest problem with this sort of inference is that there are just too many free parameters. There are Fed intentions, Congressional intentions, the effectiveness of QE, the impact of austerity, and the nature of exogenous shocks that hit the economy at the same time as the policies. So here are some interpretations of 2013 that seem consistent with a casual reading of the data:
1. Fiscal policy is very effective, but didn't really change because everyone had already expected the sequester since 2011. Monetary policy really changed, but isn't very effective. And the economy wasn't hit with any particularly big shocks.
2. Fiscal policy is utterly ineffective. Monetary policy is extremely effective, but the Fed decided to keep the U.S. at 2% inflation and a lower NGDP path than the pre-2007 trend. And who cares what exogenous shocks there were, since the Fed is so powerful.
3. Fiscal policy and monetary policy are both pretty effective. Congress decided to tighten and the Fed decided to ease in 2013, balancing each other out. And there were no big exogenous shocks.
4. Fiscal policy is much more effective than monetary policy. There was a real regime shift toward austerity in 2013. But it was canceled out by positive exogenous shocks, from the economy's natural tendency toward recovery, from the unexpected stability of Europe, from Japan's rapid growth, from shale gas, etc.
I could go on. There are more combinations.
Now you see why this is so hard, right?
As I see it, event studies are much better when done in a rigorous empirical context, where theories and hypotheses are carefully defined, dates and magnitudes are more precise, and more different observables can be brought into play. In other words, I think that this is one area where academia is clearly a lot better than blogs. (Mark Thoma agrees.)
Now that having been said, if you want a cleaner experiment in whether monetarism (in a generalized sense) is effective, look not at the U.S., but at Japan. Even there, the picture is complicated by an uncertain fiscal policy outlook, but it's a much more obvious example of a monetary policy regime shift without a simultaneous fiscal regime shift.
Mike Konczal thinks 2013 shows that fiscal policy is needed alongside monetary policy.
Posted at 2:57 PM