To show what I mean, let's take an example. Suppose I was going to try to persuade you that extended unemployment insurance has big negative effects on employment. But suppose I could only show you one academic paper to make my case. Which of these two papers, on its own, would be more convincing?
Paper 1: "Optimal unemployment insurance in an equilibrium business-cycle model", by Kurt Mitman and Stanislav Rabinovitch
Abstract:
The optimal cyclical behavior of unemployment insurance is characterized in an equilibrium search model with risk-averse workers. Contrary to the current US policy, the path of optimal unemployment benefits is pro-cyclical – positively correlated with productivity and employment. Furthermore, optimal unemployment benefits react nonmonotonically to a productivity shock: in response to a fall in productivity, they rise on impact but then fall significantly below their pre-recession level during the recovery. As compared to the current US unemployment insurance policy, the optimal state-contingent unemployment benefits smooth cyclical fluctuations in unemployment and deliver substantial welfare gains.
Some excerpts:
The model is a Diamond–Mortensen–Pissarides model with aggregate productivity shocks. Time is discrete and the time horizon is infinite. The economy is populated by a unit measure of workers and a larger continuum of firms...Firms are risk-neutral and maximize profits. Workers and firms have the same discount factor β...Existing matches [i.e., jobs] are exogenously destroyed with a constant job separation probability δ...All worker–firm matches are identical: the only shocks to labor productivity are aggregate shocks...[A]ggregate labor productivity...follows an AR(1) process...The government can insure against aggregate shocks by buying and selling claims contingent on the aggregate state...The government levies a constant lump sum tax τ on firm profits and uses its tax revenues to finance unemployment benefits...The government is allowed to choose both the level of benefits and the rate at which they expire. Benefit expiration is stochastic...
Paper 2: "The Impact of Unemployment Benefit Extensions on Employment: The 2014 Employment Miracle?", by Marcus Hagedorn, Iourii Manovskii, and Kurt Mitman
Abstract:
We measure the aggregate effect of unemployment benefit duration on employment and the labor force. We exploit the variation induced by Congress' failure in December 2013 to reauthorize the unprecedented benefit extensions introduced during the Great Recession. Federal benefit extensions that ranged from 0 to 47 weeks across U.S. states were abruptly cut to zero. To achieve identification we use the fact that this policy change was exogenous to cross-sectional differences across U.S. states and we exploit a policy discontinuity at state borders. Our baseline estimates reveal that a 1% drop in benefit duration leads to a statistically significant increase of employment by 0.019 log points. In levels, 2.1 million individuals secured employment in 2014 due to the benefit cut. More than 1.1 million of these workers would not have participated in the labor market had benefit extensions been reauthorized.
Some excerpts:
[W]e exploit the fact that, at the end of 2013, federal unemployment benefit extensions available to workers ranged from 0 to 47 weeks across U.S. states. As the decision to abruptly eliminate all federal extensions applied to all states, it was exogenous to economic conditions of individual states. In particular, states did not choose to cut benefits based on, e.g. their employment in 2013 or expected employment growth in 2014. This allows us to exploit the vast heterogeneity of the decline in benefit duration across states to identify the labor market implication of unemployment benefit extensions. Note, however, that the benefit durations prior to the cut, and, consequently, the magnitudes of the cut, likely depended on economic conditions in individual states. Thus, the key challenge to measuring the effect of the cut in benefit durations on employment and the labor force is the inference on labor market trends that various locations would have experienced without a cut in benefits. Much of the analysis in the paper is devoted to the modeling and measurement of these trends.
The primary focus of the formal analysis in the paper is on measuring the counterfactual trends in labor force and employment that border counties would have experienced without a cut in benefits...The first one...allows for permanent (over the estimation window) differences in employment across border counties which could be induced by the differences in other policies (e.g., taxes or regulations) between the states these counties belong to. Moreover, employment in each county is allowed to follow a distinct deterministic time trend. The model also includes aggregate time effects and controls for the effects of unemployment benefit durations in the pre-reform period...The second and third models...reflect the systematic response of underlying economic conditions across counties with different benefit durations to various aggregate shocks and the heterogeneity is induced by differential exposure of counties to these aggregate disturbances.
These two papers have results that agree with each other. Both conclude that extended unemployment insurance causes unemployment to go up by a lot. But suppose I only showed you one of these papers. Which one, on its own, would be more effective in convincing you that extended UI raises U a lot?
I submit that the second paper would be a lot more convincing.
Why? Because the first paper is mostly "theory" and the second paper is mostly "evidence". That's not totally the case, of course. The first paper does have some evidence, since it calibrates its parameters using real data. The second paper does have some theory, since it relies on a bunch of assumptions about how state-level employment trends work, as well as having a regression model. But the first paper has a huge number of very restrictive structural assumptions, while the second one has relatively few. That's really the key.
The first paper doesn't test the theory rigorously against the evidence. If it did, it would easily fail all but the most gentle first-pass tests. The assumptions are just too restrictive. Do we really think the government levies a lump-sum tax on business profits? Do we really think unemployment insurance benefits expire randomly? No, these are all obviously counterfactual assumptions. Do those false assumptions severely impact the model's ability to match the relevant features of reality? They probably do, but no one is going to bother to check, because theory papers like this are used to "organize our thinking" instead of to predict reality.
The second paper, on the other hand, doesn't need much of a structural theory in order to be believable. Unemployment insurance discourages people from working, eh? Duh, you're paying people not to work! You don't need a million goofy structural assumptions and a Diamond-Mortensen-Pissarides search model to come up with a convincing individual-behavior-level explanation for the empirical findings in the second paper.
Of course, even the second paper isn't 100% convincing - it doesn't settle the matter. Other mostly-empirical papers find different results. And it'll take a long debate before people agree which methodology is better.
But I think this pair of papers shows why, very loosely speaking, evidence is often more powerful than theory in economics. Humans are wired to be scientists - we punish model complexity and reward goodness-of-fit. We have little information criteria in our heads.
Update: Looks like I'm not the only one that had this thought... :-)
Also, Kurt has a new discussion paper with Hagedorn and Manovskii, criticizing the methodology of some empirical papers that find only a small effect of extended UI. In my opinion, Kurt's team is winning this one - the method of identifying causal effects of UI on unemployment using data revisions seems seriously flawed.
Update: Looks like I'm not the only one that had this thought... :-)
@Noahpinion history of thought: impetus for writing the empirical papers came from "unconvinced" referees at the QJE on optimal one in 2011— Kurt Mitman (@SorryToBeKurt) May 23, 2016
Also, Kurt has a new discussion paper with Hagedorn and Manovskii, criticizing the methodology of some empirical papers that find only a small effect of extended UI. In my opinion, Kurt's team is winning this one - the method of identifying causal effects of UI on unemployment using data revisions seems seriously flawed.
One is about optimal policy and the other about actual policy. How do you consider optimal policy empirically?
ReplyDeleteEasier to think about optimum policy once you actually understand the effects of current policy isn't it?
DeleteI agree with Donk. There are a million theories. Tweak one or two of the copious assumptions in a paper like Mitman & Rabinovitch, and you can easily flip the results on their head. For example, see this paper: http://ceg.berkeley.edu/research_27_4224264158.pdf
DeleteThe only thing that can tell you which theory to believe (if any) about the optimal policy is...empirics.
The economy is populated by a unit measure of workers and a larger continuum of firms...Firms are risk-neutral and maximize profits.
ReplyDeleteLol.
That's an unfair comparison of a theory paper using advanced methods to improve the estimates of the basic model with an empirical paper looking for a simple effect.
ReplyDeleteA fair comparison would be the basic econ 101 model (or a just slightly fancier version) predicting unemployment insurance would decrease employment (and even that reasonable priors might suggest this effect would be non-negligible) against the empirical paper you mention.
In this contest I think the theory wins hands down over the empirical work. I mean look at empirical work in other areas like gun control where true randomization is impossible and many confounders exist. Only an overwhelming consensus of empirical results produced using a variety of natural experiments, statistical assumptions and contexts is enough to really shift the beliefs of sophisticated readers who are aware of the many statistical difficulties and potential for bias. On the contrary, the basic econ 101 style argument (bolstered by more sophisticated models but with quickly diminishing returns) made this effect plausible enough to prompt all these studies.
Indeed, I suspect virtually no one would be convinced by a study showing unemployment insurance substantially decreased unemployment without at least a plausible theoretical explanation fitting the gathered data for how that might happen.
-----
Of course, this might just be a terminological dispute. I'm claiming that, because we bring along fairly informative priors into our scientific inquiry, basic theoretical explanations are more compelling than basic experimental results in fields where randomization and replication difficult and confounders numerous.
However, since the low hanging fruit always gets picked first and returns diminish quickly with increasing model complexity you are no doubt correct about the relative evidental value of modern empirical papers compared to modern theory papers given the long tradition of theoretical work and the ever present need to replicate basic experimental results.
While I generally agree with you Noah, I think the results of the competition fundamentally depend on definition.
ReplyDeleteOne could argue, for example, that by selecting unemployment benefits and employment as two concepts to spend time comparing, an author has implicitly included a fair bit of theory in his / her analysis (both in the construction of the two concepts themselves, and in the application of an economic framework where people respond to financial incentives).
Similarly, any collection and application of 'robust' quantitative data (e.g. deciding that induced variation is a valid methodology) makes use of a wealth of statistical theory.
Perhaps another way to frame the result is that theory faces more rapidly diminishing marginal returns than evidence?
"Darwin and a fellow scientist were searching for fossils in the north of England. They were not aware of the glacial theory at the time. Years later Darwin revisited the area, and he was now astonished to discover how clearly marked were the glacial ridges on the rocks. He had not noticed them on his earlier visit because he was not looking for them.... Darwin was able to appreciate the glacial markings only after he became aware of the glacial theory."
ReplyDelete-- Lionel Ruby, "How the Scientist Thinks."
One needs theory to put the empirical ducks in a row.
"Humans are wired to be scientists - we punish model complexity and reward goodness-of-fit. "
ReplyDeleteI think you may be going back on your priors on this one. Evidence, like logic, is not something humans have innately, it's something that has to be learned through education and practice, and is something done largely for the purposes of winning an argument (we are all lawyers after all). Otherwise, why do you think so many people still believe in creationism and conspiracy theories? To use an Economics example, why are Rational Expectations and Permanent Income Hypothesis still given credence despite its lack of empirical foundations?
I think that people have a deep, powerful need to feel like they understand the world. When data isn't available, they rely on theory. But when evidence does become available, I think they pretty quickly switch over.
DeletePeople use theory to organise the world so that they can act within it. It takes a lot of training to get people into a mindset where they can abandon large theories if confronted with contrary evidence - and even then it's often a slow and partial retreat rather than a switch. Given that evidence accumulates slowly and is often ambiguous, but that action is imperative, theories can outstay evidence by a very long way. If people are quick to absorb evidence and switch theories, it's usually a sign that the subject is of little importance to them.
DeleteThe second paper failed to mention that job openings spiked in 2014. Unless the increased supply of labor (due to the loss of unemployment benefits at the end of 2013) caused an increase in the demand for labor (the spike in job openings), then I would conclude that the spike in job openings not the termination of unemployment benefits caused the drop in unemployment in 2014.
ReplyDelete+1,000. Only a brain-dead idiot or an economist (but it appears I repeat myself) could conclude that UI--which is a fraction of previous earnings, capped at a low level--keeps people from finding jobs.
DeleteOf course, only someone who is brain-dead enough to claim that the drop in U-3 (read: people actively looking for work) that accompanies people discouraged enough to stop looking for work (and broke enough that they cannot after those Incredibly Rich Benefits stop being paid) counts as a "decrease in the unemployment rate."
Geez, Noah, you used to be better than this. If you can't find something that shows the E/P ratio going up after UI expires instead of something that is targeted to count discouragement as a good thing, you shouldn't be actively spewing horseshit about how UI "keeps people from working."
-1000. I must be a brain dead idiot because I know that UI discourages people from finding jobs. "A fraction of previous earnings" for doing NOTHING can be quite compelling versus working full-time. This is especially true given that finding previous earnings for full-time work after being laid off is often not such an easy task.
DeleteThere has also been a fair amount of work done, outside of the natural experiment years discussed above, to show the high correlation between UI benefits ending and new work being found.
" 'A fraction of previous earnings' for doing NOTHING can be quite compelling versus working full-time."
DeleteNot really, because it turns out people don't WANT to do nothing -- they want meaningful work. And sure, when benefits run out, you will find people tend to find something -- but this is simply desperation driving them to settle for crappier jobs well below their skill level, which comes with high costs.
How could you possibly know what all people WANT? I've sat at a UI office, and I heard people explicitly stating they got a job at McDonald's so they could work for a short time and restart their UI benefits. The arrogance of some people on the Internet is staggering, whether Anonymous or the author of a crappy blog like this one.
DeleteI never said people want to do nothing; I said they get paid for doing nothing. The distinction is that the nothing can include doing pretty much whatever you want to do outside of paid work. Personally, if my employer, or any insurance program for that matter, offered me 60% of my current wages to no longer work for pay, I'd take the deal in a heartbeat. I certainly wouldn't be doing nothing with my time, but one thing I can assure you I wouldn't be doing is looking for paid work.
DeleteExcept, since unemployment is a fraction of usual earnings, most people cannot "do whatever they want" when on it. If lucky, and a lot of people on unemployment aren't, unemployment will cover the basics leaving the unemployed stuck at home with no money to do anything else, except for maybe look for work.
DeleteI see your point, Noah, but in the case of unemployment insurance, I learned a lot from theory. Not the theory of Mitman and Rabinovitch, but the theory presented by Larry Summers: "Some Simple Economics of Mandated Benefits," The American Economic Review, Vol. 79, No. 2, . . . (May 1989), pp. 177-183. I found a link to the paper on Marginal Revolution: http://marginalrevolution.com/marginalrevolution/2009/07/some-simple-economics-of-mandated-benefits.html .
ReplyDeleteThanks. Yeah, I learned that one in grad school...not sure how right Summers is, though.
Delete"Unemployment insurance discourages people from working, eh? Duh, you're paying people not to work!" but "Unemployment insurance payouts are spent, sales revenue rises, which encourages firms to hire." I couldn't tell whether either paper takes this aspect of the matter into account. Did sales revenue decline when benefits were cut?
ReplyDeleteThese two papers represent two quite different exercises. In one, a structural model is set up and used to tell you what an optimal unemployment insurance program should look like. In the other, the data is used in a creative way to tease out the quantitative response to a particular policy action. The first paper does indeed answer the question that the second paper sets out to answer, but the second paper doesn't answer all the questions the first one does. For example, the second tells you nothing about optimal policy, which of course we can't address without a fully-specified model.
ReplyDeleteWhich approach is "better?" Why would we even want to ask that question? Each piece of research contributes something in its own right. One approach doesn't somehow "win" over the other. You can in fact learn more from doing things in different ways.
Could we say that the second approach is better because it is somehow model-free? Of course not. If you go through that paper in detail, you'll see that it requires many heroic assumptions, just as is the case for the first paper.
Finally, your criterion for making a choice between these approaches seems to be what convinces Noah Smith. But both papers were written by Mitman and coauthors - they're all part of the same research group. They're all well-trained, and more-or-less agree on research strategies. Why should any of them care whether you like the way they do their research in labor economics?
Finally, your criterion for making a choice between these approaches seems to be what convinces Noah Smith. But both papers were written by Mitman and coauthors - they're all part of the same research group. They're all well-trained, and more-or-less agree on research strategies. Why should any of them care whether you like the way they do their research in labor economics?
DeleteIf the main criterion for good research was the opinion of the researcher, all research would be good, and society's only proper response would be to give all researchers more and more money and prestige, to compensate them for all the good they're doing...
...hmm, wait a second... ;-)
"If the main criterion for good research was the opinion of the researcher..."
DeleteOf course not. 95% of what we do is convincing our colleagues in the profession. In fact, I've seen both of those papers presented - in both cases in more than one venue. There were plenty of questions, and people in the room hashed over the methods and the results. In general, we bounce ideas around in the profession, and spend some of our time trying to get those ideas across to people outside the profession.
In your post, you are trying to make a judgment about the relative quality of work that you don't know much about. Your criterion is: "Which of these two papers, on its own, would be more convincing?" But the papers weren't written to convince you, they were written to convince people who work on labor economics. If the authors were trying to convince you, they would have written something else.
¯\_(ツ)_/¯
DeleteI think you have this view of research communities as totally self-judged operations - that people outside a research community should just accept that the people in the community are doing good and valuable stuff, and therefore continue to subsidize them with money and prestige. But I think at some point people from outside are naturally going to look in and say "Hey, what am I getting out for this money that I'm putting in?"
In natural science, the payoff is at least comprehensible - eventually, science often gets turned into useful technology. With economics, the output is usually policy advice, and people want to know that they're getting good policy advice for their money instead of some silly stuff. And I think that evidence is generally much more powerful than theory at convincing people that the advice they're getting is good. Also, I think evidence should be more powerful, so I'm advocating for that.
Economists cost money. Given their high salaries - usually much higher than natural scientists - it makes sense for society not to simply assume that it's always getting full value for that investment.
"But the papers weren't written to convince you, they were written to convince people who work on labor economics. If the authors were trying to convince you, they would have written something else."
DeleteAnd there's the statement of some truly clueless about the real value of knowledge and who obviously spends way too much time in his own little bubble (which would also explain some of the gross ly ridiculous claims he has made, but that's a book in and of itself).
In other words: get a clue. No wonder economists are laughed at outside their profession (and only have their salaries because they are the handmaidens of big business).
Again, Noah, these papers were not written to justify economic methods to non-practitioners. For either, you could make a good case that this is good science, and why the average person on the street should care about it. In any case, I'm glad you're not in a position to allocate resources in the economics profession, as you don't know as much as you think you do.
Delete"Written to justify methods"...not sure I even know what that means!
DeleteAs for allocating resources in the econ profession, it's interesting to ask who does that job. It's not granting agencies. It seems to be mostly the administrators of liberal arts schools at universities, and the government through the Fed system. I wonder how administrators make their decisions about how much money to pay economists. Obviously economists are going to clamor for more money, but I'm sure administrators know they're going to do that and react accordingly. A lot is probably determined by the demand for econ classes, but I guess it's possible to find people willing to teach econ classes relatively cheaply. Why do we have so many econ professors sitting around in econ departments getting paid very high salaries to do research? The research is not sold for a price, so how is marginal benefit calculated? Hmm.
Oh, and Steve, if there's stuff you think I should know that I don't know, stop wasting time telling me how much I supposedly don't know, and just tell me the stuff, so then I'll know it.
DeleteHe isn't arguing that you don't know something. He is arguing that you are taking papers out of context.
DeleteIt seems like he is trying to say that the two papers were written for targeted audiences, while you seem to be judging them as if they both were written for a general audience. Perhaps you should not be judging them with the same rubric.
I agree with the believability of the papers.
ReplyDeleteI don't think anyone prominent every claimed that unemployment insurance didn't hold unemployment higher than otherwise. I just think people felt it was the ethical thing to do to extend benefits. There's a lot of humanity that gets lost in the numbers. There are a lot of negative consequences that don't make it into economic papers. The assumption is that the numbers are what matters. This is a big assumption.
The numbers don't necessarily capture what it is like to be laid off from a well-paying manufacturing job, to spend a couple of years looking for a similar job and to end up working for McDonalds for minimum wage, selling the house and perhaps declaring bankruptcy in the process.
Economics without attention to humanity is a scary thing.
The optimal cyclical behavior of unemployment insurance is characterized in an equilibrium search model with risk-averse workers. Contrary to the current US policy, the path of optimal unemployment benefits is pro-cyclical – positively correlated with productivity and employment.
ReplyDeletework injury compensation
ReplyDeleteSean Callagy & Callagy Law team has won Two, 27 million dollar verdicts in 2 years. The firm focuses on litigation, medical revenue recovery, & family law in New Jersey, New York and Arizona. You can contact any time site: http://www.callagylaw.com
ReplyDeleteSean Callagy & Callagy Law team has won Two, 27 million dollar verdicts in 2 years. The firm focuses on litigation, medical revenue recovery, & family law in New Jersey, New York and Arizona. You can contact any time site: http://www.callagylaw.com
ReplyDeleteSean Callagy & Callagy Law team has won Two, 27 million dollar verdicts in 2 years. The firm focuses on litigation, medical revenue recovery, & family law in New Jersey, New York and Arizona. You can contact any time site: http://www.callagylaw.com
callagy law
Paramus law firm
sean callagy