Goodhart's Law is often called a generalization of the Lucas Critique, but it's really not. The "law" actually comes in several forms, one of which seems clearly wrong, one of which seems clearly right. Here's the wrong one:
Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.That's obviously false. An easy counterexample is the negative correlation between hand-washing and communicable disease. Before the government made laws to encourage hand-washing by food preparation workers (and to teach hand-washing in schools), there was a clear negative correlation between frequency of hand-washing in a population and the incidence of communicable disease in that population. Now the government has placed pressure on that regularity for control purposes, and the correlation still holds.
Incidentally, this also means that the oft-repeated statement that "Social science can never discover any laws of nature, because human beings react to the discovery of the 'law'" is false. For example, suppose you see that when gas prices rise, people drive less. You place pressure on that regularity for control purposes by instituting a gas tax, which raises the consumer price of gas. Lo and behold, people drive less! Sometimes people actually do what you try to get them to do.
Anyway, here's the version of Goodhart's Law that seems obviously true:
As soon as the government attempts to regulate any particular set of financial assets, these become unreliable as indicators of economic trends.This seems obviously true if you define "economic trends" to mean "economic factors other than the government's actions. In fact, you don't even need any kind of forward-looking expectations for this to be true; all you need is for the policy to be effective. In other words, this law could as easily describe "Milton Friedman's thermostat" as the Lucas Critique.
An application of this correct form of Goodhart's Law is the suggested use of financial market outcomes as guides for Fed policymaking. At a speech at a conference at Michigan last year (and elsewhere), Narayana Kocherlakota suggested that instead of using its own internal inflation forecasts, the Fed should set policy according to market forecasts of inflation. Specifically, he suggested using the prices of TIPS and other inflation-dependent financial assets to calculate risk-neutral probabilities of future inflation, and to set Fed policy accordingly. His reasoning was that unlike the Fed's own forecasts, risk-neutral probabilities take into account the degree to which people value price stability in different states of the world.
I raised an issue with this approach. The risk-neutral probabilities obtained from financial markets are unconditional probabilities - in other words, markets may have already taken into account what they expect the Fed to do, but they have not yet taken into account what the Fed actually decides to do. The Fed's ultimate policy decision is something the Fed knows and the markets do not. Thus, the Fed is able to make internal conditional forecasts for each possible policy choice, and use only one of those conditional forecasts when making its decision. If it relies on markets, it must necessarily use unconditional forecasts, which may be less informative about conditional outcomes (which is what the Fed really cares about).
This seems to me to be an application of the second version of Goodhart's Law. If you set interest rates mechanically (i.e. according to some rule) based on current market expectations of inflation, you will change those expectations, and markets will move, requiring you to set interest rates differently according to your policy rule. It's possible that markets and policy might converge to some stable equilibrium, but also possible that market volatility might increase once it was known that policy was being set based on market prices themselves.
To link this to Goodhart's Law, if the Fed targeted the prices of inflation-linked assets, those prices would mostly contain information about expectations of Fed policy decisions (which the Fed already knows better than the asset market), rather than about non-Fed economic forces that might affect inflation or people's utility of stable prices (i.e. the things the Fed wants to use the asset prices to ascertain).
So Goodhart's Law, in its obviously true form, seems very important for policymaking.
Noah, I agree with almost everything here...except for the last point about internal and market expectations. Here's why. Suppose that the central bank set a nominal interest rate in period 0 according to the rate of inflation in period 1 and expectations for future periods. Let's call this i(pi_m). Suppose that, in period 0, the interest rate was different and set by some other rule, or perhaps not one at all, so i_0. Let's assume that the policy rule causes market expectations of inflation to change.
ReplyDeleteWhat you're saying is that we have a circularity/recursion problem, in essence, between i and pi_m because the two are dependent on the other. But couldn't you have i and pi_m work in a converging alternating series over multiple periods?
Also, sorry for the double comment, but that seems to me to be the way that this would work out in real life, that the markets take the limit of the Fed's responses to inflation expectations and inflation expectations' responses to monetary policy.
DeleteDouble comments are grudgingly permitted in this comment section... ;-)
DeleteAnyway, yes, this could happen, but the "ringing" effects would be inefficient, and that inefficiency might well outweigh the information loss from the Fed using its own internal forecasts.
As for markets "taking the limit" of the Fed's responses, keep in mind that the market can't know which way the Fed will break, while the Fed can and does know this. There's always an information asymmetry here. Kocherlakota was calling for the Fed to base its policy on the current inflation expectations of an actor with potentially much less information than the Fed itself.
One thing to consider: the Fed (as in the FOMC) does not really have an official forecast. You can see in speeches and other official documents that there are often differences of opinion among the policy makers. For example, the Summary Economic Projections http://www.federalreserve.gov/newsevents/press/monetary/20130320b.htm summarizes the range of views. Even if you have lots of information, you have to know how to combine it and communicate it and predict the reaction ... many interesting questions here.
DeleteGreat points! How did Narayana reply?
ReplyDeleteHe said "That's interesting, I'll have to think about that..."
DeleteSounds a little like the parable of the Ox, a more flippant look at a similar issue: http://www.johnkay.com/2012/07/25/the-parable-of-the-ox
ReplyDeleteI'm trying to think about what the true version of Goodhart's Law would imply for the NGDP targeting idea of creating a NGDP futures market and having the Fed conduct open market operations to achieve a target price for these claims. Is that idea supposed to work because of Goodhart's Law, or in spite of it?
ReplyDeleteBasically, that hinges on the question of whether the Fed is "omnipotent" (i.e. if it can hit any NGDP path it wants), as Sumner and Beckworth assert. If so, and if people believe that it's so, then the NGDP futures market will never vary from the path the Fed decides on; all of the information contained in the target price will be information about beliefs about future Fed actions, but that's OK, because future Fed actions = future reality.
DeleteIf the Fed is not omnipotent, then the futures price will be a poor target, as most of the information contained in the price will be people's beliefs about Fed actions, and the Fed will have to change its actions to reflect those beliefs, causing greater uncertainty about future Fed actions and increasing the negative feedback loop (imagine NGDP futures as Bitcoin). That would also cause fluctuations in the real economy, due to the real (though not omnipotent) effect of Fed policy.
Basically, my intuition is that the former case is vanishingly unlikely, and that we'd see "Goodhart instability" rather than "Goodhart stability" of an NGDP futures market target, causing the target to be abandoned by the Fed.
If you make a distinction between causal variables and correlated variables, it becomes clearer when Goodhart's law is true. Your hand washing example shows that if the variable A causes variable B, controlling A changes B. But if, for example, people bought more ice cream on hot days, you can't control the weather by outlawing ice cream.
ReplyDeleteYep. Or in Lucas terms, you have to figure out which variables are "structural".
DeleteNoah, You are still misinterpreting NGDP futures. It has nothing to do with a policy where the Fed looks at NGDP futures and adjusts policy accordingly. That policy was criticized by Bernanke and Woodford (JMCB, 1997), where they referred to the "circularity problem." I'm not sure if the circularity problem is that worrisome in practice (I'm inclined to agree with Evan Soltas) but even if it is it has no bearing on NGDP futures targeting.
ReplyDeleteThe way to think about NGDP futures targeting is to expand the FOMC from 12 to 100 million. And then pay FOMC member salaries based on the accuracy of their policy decisions. Obviously neither of those changes to the FOMC runs into any sort of "Goodhart's Law" problem, and hence NGDP targeting is in no way affected by Goodhart's Law.
It has nothing to do with a policy where the Fed looks at NGDP futures and adjusts policy accordingly. That policy was criticized by Bernanke and Woodford (JMCB, 1997), where they referred to the "circularity problem."
DeleteYes, that's what I was thinking of.
The way to think about NGDP futures targeting is to expand the FOMC from 12 to 100 million. And then pay FOMC member salaries based on the accuracy of their policy decisions.
This is incredibly vague, and does not describe how the policy would actually be implemented.
Scott had a pretty specific blog post here:
Deletehttp://www.themoneyillusion.com/?p=1210
Bill Woolsey discussed Scott's alternative trading scenarios in gory details here:
http://monetaryfreedom-billwoolsey.blogspot.com/2010/12/sumner-and-delong.html
The idea behind NGDP futures targeting seems to be to add base money (or do anything else really) until the futures contract price equals the target NGDP level. In theory, this creates information symmetry as both markets and the Fed will know what the Fed will do if the contract price deviates from target. Taken to its ultimate conclusion, the Fed would never need to act since speculators wouldn't bet against them, the contract price would always equal the target level, agents would behave in accordance with the future's forecast of NGDP and so produce it exactly, NGDP would be spot-on trend, etc.
DeleteSeems to me the problem is not so much Goodharts Law but the Fed's ability to hit short term (realized) NGDP targets. Market Monetarists rely heavily on Fed credibility (expectations setting) to make this problem go away. If credibility is lacking, it can produce quite stupid results. For instance, if markets clearly see that NGDP will fall short, but the Fed insists on maintaining its target, the Fed could end up, say, owning all Treasuries in a failed "command the tides" operation. Theoretically, this mega-QE could happen in a matter of weeks before contract expiration. Imagine the disruption to asset markets. If the Fed adds the ability to allow contracts to expire below/above target, this would only further harm their credibility.
The best formulation is not, according to La Wik, due to Goodhart, but rather to Marilyn Strathern:
ReplyDelete"When a measure becomes a target, it ceases to be a good measure."
Point being that this measure need not be financial. Take an example from the field of education. In the UK, the government places huge emphasis on certain outcome measures, like particular grades in GCSE and A Level exams. So the whole edifice is geared towards producing these measures. But what does "number of students achieving grade A-C" (or whatever) actually represents in such a scenario? What it represents is how successful the education system is in producing students of grade A-C. But this is circular.
Delete"As soon as the government attempts to regulate any particular set of financial assets, these become unreliable as indicators of economic trends."
ReplyDeleteThat seems to assume that market valuations of financial assets actually can serve as reliable indicators of economic state or trends without government regulation, but history has long demonstrated that this is false. Financial markets in the 19th century were only lightly regulated, but they followed a chaotic boom and bust cycle pattern which had little to do with underlying values or the actual state of the economy.
Of course, there was some government regulation. For example, you couldn't kill your financial rivals in cold blood and steal their assets without some government involvement if you did it often enough. Perhaps this was sufficient to make financial assets unreliable indicators.
we do read in downunder noah and we get your noahopinion!!
ReplyDeletekeep up these great articles
I always understood Goodhart to mean that if you want a target, it's best to choose a final goal target, because targeting an intermediate will change the relationships between the intermediates.
ReplyDeleteyou don't fly an aeroplane by looking at the position of the controls.
This comment has been removed by the author.
ReplyDeleteIf you target hand-washing, reward washers and penalize non-washers sufficiently, hand-washing will get misreported.
DeleteHow well would NGDP targeting work, using the way China reports GDP?
An awful lot of money would be riding on the outcome of a number that relies on a lot of (usually well-thought-out) estimates and judgment calls on the part of the people who report it... as illustrated by the monthly revisions and often significant changes to GDP path at benchmark revision time.
If you target hand-washing, reward washers and penalize non-washers sufficiently, hand-washing will get misreported.
DeleteSure, and it does in real life! But the misreporting is only a small fraction of the actual real compliance, and public health has drastically improved as a result...can't argue with results!
wouldn't necessarily be the case if billions were riding on the reporting. sure, a moderate cigarette tax will reduce smoking, a large one (eg illegal drugs) will increase bootlegging.
DeleteThe more money is at stake and the squishier the number you're targeting, the more important Goodhart's law...It's not 2 versions of the law, it's just in a domain / regime where the reporting isn't that squishy, the policy isn't that radical, the stakes aren't that huge.
by the time you get to macro policy, the the stakes are huge and numbers are pretty squishy.
not sure if this NGDP futures argument is a thought experiment or a serious proposal. If it were the latter, I wonder if they've ever looked at how GDP is estimated.
Minnapolis Fed President Kocherlakota’s conjecture of using a policy instrument that estimates risk-neutral expectation of inflation, calculated using risk-neutral probabilities has its limits as pointed out by Noah Smith. In my article, ‘How the market ignores conditional probabilities’, I have argued that as we deal with expectations related to sequential information gathering, every additional information changes the probability of the event (conditional probability); use of Bayesian statistics and with the acceptance of shrinking sample space we could get rich insights for reasonable conditional probability estimates of posterior events based on available information of past events; Fed's unconditional approach to risk-neutral probabilities takes a limiting view.
ReplyDeleteI am sad to say, that your example with the cars, does not hold when it meets econometrics.
ReplyDeleteLast week I saw my sister defend her dissertation on cars and taxes.
Their data shows, that if you raise taxes on gas, some people buy a smaller car, and then end up driving more than before the tax was implemented. I think this must be scored as one for Goodhart's law.
Amazing besides stimulating area. Fantastic variables you could have generally distributed to us. Many thanks. Merely move forward creating this sort of write-up.
ReplyDeleteRegards: Louisville Auto Accident Attorney