Saturday, May 14, 2011

Did the stimulus really destroy a million private-sector jobs?



Hey non-economists, want to see what's inside the guts of one of those econ papers you keep hearing about? Well that's why you have grad student bloggers like me. We read the papers so you don't have to.

This week, Greg Mankiw links us to a paper by Timothy Conley of Western Ontario and Bill Dupor of Ohio State University. The paper's eye-popping finding is that the American Recovery and Reinvestment Act (ARRA), also known as the Stimulus, was responsible for a net loss in jobs. No, really! From the paper's abstract:
Our benchmark results suggest that the ARRA created/saved approximately 450 thousand state and local government jobs and destroyed/forestalled roughly one million private sector jobs...The majority of destroyed/forestalled jobs were in growth industries including health, education, professional and business services.
Wow! Seriously? The stimulus directly resulted in a net loss of five hundred and fifty thousand jobs? That's it, I'm voting Republican from now on...

But wait. A still small voice is nagging me from the back of my mind, urging me to read beyond the abstract. And so my dissertation will have to wait 40 minutes while I wade through three dozen pages of PDF in search of an answer to my nagging doubts.

Because I do have some doubts about this result. Stimulus spending destroys jobs? How the heck is that supposed to work? I mean, maybe you believe in full Ricardian Equivalence, but that would just predict that stimulus is a wash. Perhaps people are cutting back spending in anticipation of the deadweight losses caused by the future taxes needed to pay back the stimulus-related borrowing*? Hmm, maybe, but that sounds so preposterous that I kind of expected something to be fishy about this paper from the get-go.

What Conley and Dupor do is to run a state-by-state regression. Different states received different amounts of ARRA spending, so looking at the differences in employment growth rate between those states after the passage of ARRA should tell us how many jobs ARRA created or destroyed. This should lead to a regression of the type:
Employment growth = A + B*Stimulus + C*Other Stuff + e
Now, you may say: "Wait, but states where employment goes down should be expected to get more stimulus money, since those are just the states that were hardest-hit by the recession!" And you'd be right: there is a big endogeneity problem here. After all, the fact that there's a bunch of sick people in the doctor's office doesn't mean that doctors make you sick. Messrs. Conley & Dupor deal with this problem by finding some "instruments" - natural sources of variation in the amount of stimulus money that a state gets, that have nothing to do with how bad the state's economy was doing. Usually, critics of an empirical paper like this will try to say that the instruments used are bad ones - that they actually can be affected by the business cycle, or that they don't give rise to enough variation in stimulus funding. 

I am not going to do that. I am going to give Conley & Dupor a free pass on their instrumental variables, because I already see one and possibly two gaping hole in their analysis that makes the instrument problem somewhat of a sideshow.

(Update: I had initially written about a second possible problem with this paper, but commenter Ivan found evidence that (thankfully) that problem didn't exist. So, in the interests of not making people read several pointless paragraphs, I've deleted the section that was previously here. Thanks, Ivan!!)

On page 20 of their paper (Table 4), Conley and Dupor have a table that shows their main result: the number of jobs that they estimate to have been created or destroyed by the stimulus. In all private sectors, the estimates are negative. BUT, check out the confidence intervals in Table 4. With one exception, the upper limits of all the confidence intervals are highly positive. This despite the fact that they use a less-rigorous 90% confidence interval (instead of the standard 95%).

This means that Conley and Dupor's results are statistically insignificant. Bluntly, what they have found is nothing. Formally, if we use their model to test the hypothesis that the stimulus caused a net increase in private-sector jobs, we will not be able to reject the hypothesis.

Conley and Dupor tweak their model with some alternative specifications. No change. As you can see in Table 7 and Table 9 (p.23-4), upper 90% confidence limits continue to be strongly positive. If the authors really did leave the intercept term in their regression equation, then that's probably why they got insignificant results; if not, then there's some other problem with their instruments or their specification, or maybe just the data itself.

But, given the lack of any statistically significant findings, this paper does not deliver the results that it advertised. Conley and Dupor's abstract should read "We find no evidence for a significant effect of the ARRA on job creation." That would be scientifically honest, but would not turn a lot of heads. Instead, the abstract makes the more politically incendiary claim that the ARRA destroyed jobs, which the authors actually did not find. They do leave themselves an escape rout by using the word "suggest," but I am not satisfied. In my opinion this is a paper that overstates its findings. (Note: Conley and Dupor have since revised their abstract significantly to more accurately reflect their results, for which I commend them!)

My guess is that papers like this get attention because of politics, not because of science. Dr. Mankiw linked to this paper without comment, evaluation, or qualification. But he could have just as easily linked to this paper by Daniel J. Wilson, which uses a methodology similar to that of Conley and Dupor, but finds strongly positive (and often strongly significant) effects of the stimulus.


Update: Arnold Kling is also not a fan of Conley-Dupor


* Actually, it's worse than that. You have to also assume that future deadweight losses from stimulus-payback taxation will be highly concentrated in the states that received the most stimulus funding; i.e., that taxes will be specifically targeted at those states! 

32 comments:

  1. I looked over the paper only quickly, but it seems that they do use intercept (page 12, bellow eq. 3.1):

    "Here ANC is a column vector of state-specific ancillary regressors: two lags of employment growth, the maximum monthly unemployment insurance payment, region dummies, state population and a constant."

    But otherwise yeah, the fact that they ignore uncertainty of their estimates sounds fishy.

    ReplyDelete
  2. Very nice analysis. I thought the fact that they had such huge confidence intervals to be very strange as well.

    ReplyDelete
  3. When I noticed Mankiw's post I glanced at the paper just to see if it was worth my time reading. I could see immediately that there's a much more fundamental problem with this approach and it applies to Wilson's paper (and Fayrer and Sacerdote's as well).

    State level studies tell us nothing about the macro effects of fiscal stimulus. It’s a near perfect example of the fallacy of composition. If the federal government builds a billion dollar military base in Fargo, North Dakota, I think most economists would agree (Conley and Dupor excepted) that the number of jobs increases in Fargo, North Dakota. Does the number of jobs increase at the national level? Possibly yes, but nothing in these state level studies actually addresses that question.

    State level studies can’t tell us whether fiscal stimulus works just as state level studies can’t tell us whether monetary stimulus works. The Conley and Dupor paper is, just as I imagined, simply the worst of a rotten lot. It only gets attention because of its eye popping abstract.

    And that's the only reason why Mankiw posted it.

    ReplyDelete
  4. I find it interesting that they claim that amongst the jobs lost were professional services. As a civil engineer whose job loss was forestalled by a stimulus project, only to get laid off when the local government budgets tightened, I can't see how the sizable job losses in our industry could possibly be blamed on the stimulus.

    ReplyDelete
  5. Private sector health and education jobs? Perhaps we could just issue contracts to all government employees and decrease government jobs and increase private sector jobs at the same time with a wave of the hand.

    ReplyDelete
  6. @Mark

    http://www.themoneyillusion.com/?p=9056

    Busted. Unless you're actually behind that other blog. In which case, apologies.

    ReplyDelete
  7. Adam,
    No need to apologize, you got me. I occasionally synthesize comments by borrowing phrases. I'm much more careful on things I actually present as my own work (i.e. research papers).

    I don't think Scott minds as he knows me pretty well (I was a frequent commenter there before Scott took a break from posting). His example of a military base in Fargo was so eloquent I stole it.

    ReplyDelete
  8. If they have wrong numbers and they're interpreting them wrong, then it would seem to me that the fundamental problem is not that they're interpreting the numbers wrong; it's that they have the wrong numbers. So let's take a moment to explain why the instruments are bad.

    The point of instrumental variables is to get around confounding. A necessary condition for this to work is that the instrument and the outcome must not be confounded themselves.

    The common causes of stimulus funding and employment changes one might worry about include wealth, income distribution, and urbanisation. Let's group these together under the heading "development", and see if they affect the proposed instruments. If they don't, they may be usable instruments.

    Will development affect highway improvement funding? Yes.

    Will development affect Federal taxes and spending? Yes.

    Will development affect the relative importance of sales tax? Yes.

    Will development affect the strength of a balanced budget rule? Yes.

    Will development affect whether the governor is a Democrat? You betcha.

    All the proposed instruments are themselves confounded. It is very difficult to think of a variable affecting stimulus funding that appears as if randomised with respect to development. This is because stimulus funding is not random.

    This means that if you want to use a regression-type approach, you need to control for development and all other confounders. This may be impossible. If, however, it is possible to control for all confounders, then you don't need instrumental variables!

    This is why the instrumental variable approach -- not just the instruments used here -- is the wrong way to work on this problem, and many problems like it.

    ReplyDelete
  9. Anonymous6:03 PM

    Mankiw! His schtick during this administration has been severely annoying. He links to all manner of flat-earth crankery that seeks to admonish Democrats and Ben Bernanke, without necessarily endorsing it, but in a way that gives it implicit credibility. I take it that when he said in 2008 that Keynes was the economist we needed to turn to, he actually meant that we should ignore Keynes and also ignore Friedman and instead provisionally treat Mises as our great authority without ever formally articulating it.

    ReplyDelete
  10. @ Mark:
    Does this quote from the Conley and Dupor paper capture the essence of your (Sumner's) Military Base example?

    Page 28:
    "...allow for cross-state positive spillovers. This might result in estimates of a large positive jobs effect... Our methodology cannot pick up this effect."

    ReplyDelete
  11. Anonymous5:40 AM

    "a paper that overstates its findings"

    I'm shocked, SHOCKED...

    ReplyDelete
  12. Eric,
    Conley and Dupor are proposing to address the potential spillover effect by converting a cross state study into a panel study. (How it would capture that effect they don't bother to explain.) This might increase their estimate of jobs created but it still doesn't address the fundamental criticism that state level studies can't tell us anything about the effectiveness of a fiscal stimulus nationally (i.e. in a monetary union).

    ReplyDelete
  13. Why focus on the positive part of the confidence interval? Why not argue that the impact of the stimulus was much more negative than their estimates?

    Also, unless their standard errors are infinite (and the confidence intervals the entire real line), you are definitely wrong that estimates that are not statistically significant tell us 'nothing'.

    ReplyDelete
  14. Andy: That is a good question, and a very deep one.

    The short answer is that there are a ton of different models we could use to analyze the impact of the stimulus, and that since we don't know which one is the best model, we use statistical significance as a first-pass guide to tell us when we've found something interesting. If your confidence interval is the size of Jupiter, you probably haven't used the best model.

    The long answer is about Bayesian vs. frequentist notions of probability, and is the deepest question in all of statistics. See here for a quick primer: http://www.statisticalengineering.com/frequentists_and_bayesians.htm

    I may write some blog posts about that in the future.

    ReplyDelete
  15. What is the hypothesis being tested? Isn't statistical significance or insignificance an issue only in the context of some hypothesis test?

    ReplyDelete
  16. This comment has been removed by the author.

    ReplyDelete
  17. "If your confidence interval is the size of Jupiter, you probably haven't used the best model."

    I don't agree. I can easily construct: a) an incorrect model with very small standard errors (relative to the true DGP) or b) a correct model where the standard errors are several orders of magnitude greater than the true effect size.

    I don't see how Bayes v. frequentist issues are relevant here. Conley-Dupor's approach and your critiques are 100% frequentist.

    ReplyDelete
  18. Andy: Yes, but it's your critique that's Bayesian. Think about it... ;-)

    ReplyDelete
  19. Sorry, I don't get it. I was just pointing out that the CI includes negative values as well (more of them, in fact) but your narrative was focused on the positive values.

    ReplyDelete
  20. Also, I forget that the internet is humorous sometimes. :P

    Anyway, I think the ad hoc Bayes updating from this paper is

    1. Maybe the stimulus had crowding out (more than we thought)

    2. We should be skeptical of *any* claims of the effect of the stimulus. Standard errors are likely to be too big given sample sizes and auto/spatial correlation issues.

    ReplyDelete
  21. Because of entropy, you cannot create an economic perpetual motion machine. Any theft (tax, borrowing) from the private sector to support the public sector results in a loss of economic activity in the economy.

    By definition the stimulus had to destroy jobs. Governments takes money from producers and gives it to government workers. At the very least it destroyed the value of the money it took from the private sector less the value of the government workers and the transfer costs. Since most government workers do not add value to the economy (many actively destroy economic value) and certainly do not add as much value as private sector workers in any case, the net means it had to destroy jobs.

    ReplyDelete
  22. @Dale: these are all interesting claims. Do you have any relevant research to back them up?

    ReplyDelete
  23. Noah, I agree. The data is too murky to make the claims that the stimulus actually destroyed jobs. But the headline sure does attract eyeballs.

    ReplyDelete
  24. "Why did Mankiw pick the Conley-Dupor paper for a shout-out, and ignore the Wilson paper? Does he think that papers claiming that the ARRA was effective get an inordinate amount of attention? Is he trying to make the blogosphere more "fair and balanced"? Or does he just have a bone to pick with fiscal stimulus in general?"

    Let's see - he's a right-wing hack? That answers all of those questions.

    ReplyDelete
  25. Stone Glasgow said...

    "Why does government spending as a percentage of GDP correlate with unemployment?"

    The stupid! It hurts?

    ReplyDelete
  26. I am really bothered by your comment "Conley and Dupor's abstract should read "We find no evidence for a significant effect of the ARRA on job creation." That would be scientifically honest, but would not turn a lot of heads." I know Conley very well and know few people who are more scientifically honest. The abstract says "Our benchmark point estimates suggest..." then "There
    is appreciable estimation uncertainty associated with these point estimates" and then describe the size of the confidence intervals. How can describing precisely what you find possibly be perceived as intellectually dishonest? Now perhaps you don't like how Mankiw represented it, but why take such a low shot at Conley and Dupor?

    ReplyDelete
  27. Chris - That's because Conley and Dupor revised their abstracts significantly after I wrote this blog post.

    ReplyDelete
  28. As business owners and taxpayers have to consider the impact of public expenditure in our businesses. The idea of ​​new money being airlifted to our community sounds appealing, but the unintended consequences can do more harm than good.

    ReplyDelete
  29. Anonymous5:17 PM

    Have you checked out this paper by Cohen, Coval, and Malloy?

    http://www.people.hbs.edu/lcohen/pdffiles/envaloy.pdf

    It uses older data, but shows a similar result.

    ReplyDelete
  30. This is a great blog. I just thought I'd mention it.

    ReplyDelete
  31. Noah, I agree. The data is too murky to make the claims that the stimulus actually destroyed jobs. But the headline sure does attract eyeballs.

    ReplyDelete