Wednesday, March 15, 2017

Misuses of empirical econ

Tyler Cowen has a new post in the ongoing blog discussion about the value of empirical economics. Most of Tyler's post is about the potential misuse of empirical economics. He writes:
The political process does not select for humble versions of empiricism.  Those end up with virtually no political influence, whereas some of the more dogmatic form of empiricism may find some traction.
Absolutely true. Ideologues just pick and choose results that support them. If 100 studies show the impact of immigration on labor markets is small, and one George Borjas study says it's large, the anti-immigrant people will wave around the Borjas study. (And then the anti-empiricists will say "See? No one can really know who's right!")

A lot of the bias in empirical methods comes simply from which questions are asked/answered. Post Trump and De Vos, I see plenty of commentators and researchers reporting “vouchers don’t raise test scores” and virtually no “vouchers increase parental satisfaction.” Is that empiricism? In isolation, maybe. In terms of reflecting the broader spirit of science, not so much. It is also not humility.
This is certainly true. I'm not sure focusing on certain questions and forgetting about others signals a lack of humility. Groupthink, maybe, but not arrogance.

I find a very common pattern among both researchers and commentators.  They first form...judgments about social systems, based on overall views of history, current politics (too much)...They then view very particular empirical debates through the broader lenses they have chosen.  For instance, views on politics used to correlate with views on the interest elasticity of money demand.  Today views on politics correlate with views on minimum wage elasticity, and so on.
Yep, this is certainly going on all over the place. There's evidence for ideologically motivated reasoning in econ research, though I think the evidence doesn't show that much motivated reasoning. The problem is almost certainly worse for us commentators, and worst of all for politicians. In general, the less people know about how econ research works, the more they seem to pick and choose the results they like based on which ideology it seems to support. Stopping people from doing this is a Sisyphean task for those of us commentators who are dedicated to a more dispassionate analysis of the facts, and even we aren't always the good guys either.

BUT, consider the other approaches to understanding the world. There's formal theory. There's intuition formed from exposure to theory ("thinking like an economist"). And there's intuition formed from exposure to casual observation and stylized facts (which Tyler calls "relatively general empirical judgments").

I'd argue that all of these are equally or more likely to be misused by commentators, ideologues, and politicians in exactly the same way Tyler describes empirical results being misused. People who want to justify fiscal austerity will wave around DSGE models that support fiscal austerity. People who want to kill the minimum wage will use theoretical intuition to claim that minimum wage hurts employent ("Demand curves slope down, DUH!", etc.). People who want industrial policy will use stylized facts - which Tyler calls "broad empiricism" - to point out that development successes like Korea and Japan usually have a lot of industrial policy in their past. And so forth.

The point is: Ideologues gonna ideologue. No economic analysis method will completely or even mostly put a stop to motivated reasoning. The hope for empirical economics is that a weak signal of reality is better, over the long term, than no signal at all. Over time, the hope is that the solid studies push out the shaky ones, that economists' natural rationality and desire to know the truth inches the academic consensus toward a better correspondence with the facts.

I'm optimistic - I think the empirical revolution in econ is no fad. Yes, there will be setbacks, as hot new techniques are over-applied, and as prominent results fail to replicate. But even better techniques will be developed, meta-analyses will weed out spurious results, and armies of smart, fair-minded grad students will comb through methodology sections and data sets to separate the wheat from the chaff. Eventually, theory will follow the weak but insistent tug of evidence - theories that don't fit the facts so well will gradually fall into disuse, while those that explain the most solid results will inch into the limelight. Classes will teach these more popular theories, and the next generation of econ majors will learn intuition that corresponds just a little more closely with observable fact. Eventually, people who rely on the "broad empiricism" of casual observation will start noticing the stylized facts that agree with their new, better theoretical intuition. Thus, progress will crawl forward, bit by bit, never reaching truth, but always headed in more or less the right direction.


  1. I think it can be very useful when people hold incompatible beliefs they haven't realized or recognized, helping them become more careful and consistent.

  2. Empirical economics seems like it could use something we find necessary in medical research- the meta analysis - comprehensive review of different studies to look for the overall likely effect.

  3. I know you think that the practice of economics provides value. In a way I wish I could share your optimism. If it's going ever be more than a bunch of questionable models based on a bunch of oversimplified assumptions, it has to be useful in practical terms. And as long as economists themselves can't come to agreement on behavior and outcome, then what are the rest of us to do? Why should we care at all? This failure ends up transferring power to those politicians and pundits who are unwilling to accept the fact that "it's complicated."

    I get that economists may enjoy writing papers and articles, but unless the math is truly illuminating, rather than in the end obscuring, it ends up an academic exercise. Economists tend to get lost in numbers, because numbers are things that one can analyze and model. In doing so, they often lose sight of real resources on which lives are based. Economists tend to see the world as it is, not as it could be. Perhaps economists should do more brainstorming and less analyzing. Perhaps they'd have more success.

    (Note: It's not just economics. Medicine is littered with failed theories and treatments. Some systems are just way more complex than we seem willing to admit.)

  4. Quite frankly, I don't Tyler makes a good case against empiricism. Just ah hoc examples.

  5. Bill Ellis3:44 PM

    The Institution of economics is Intuition...Empiricists are the "barbarians" at the gate...

    It sounds a lot like what always happens... a theological institution rejects science and real world experience as dangerous and uninformed...

  6. I'm really glad to see the rise of empirical research in economics being taken seriously as a means for developing better theories, rather than simply backing particular arguments.

    Next on the program, I'd like to see economists take a good look at accounting which has provided a mechanism for assessing economic entities for centuries and fairly effectively. Too many economic theories require economic entities to do things that are untenable under the rules of accounting. For example, they often assume people will spend money they don't have without having to take out a loan or selling something.

    One step at a time, and economics could turn out to be a useful field of study.

  7. Vitor Margato1:26 PM

    In rough ideological terms, I'm probably more aligned with Tyler Cowen, Russ Roberts and John Cochrane than with you. That said, I think your contribution to that debate about the role of empiricism in economics is extremely valuable. You're right -- Russ is not. You're right, Autor's right, Angrist's right, all those folks doing modern economics in a remarkably competent and rigorous fashion are right.