Sunday, February 14, 2016

More on empirics in Econ 101


Unfortunately, I see that my friend Allison Schrager has misinterpreted my proposal for reforming Econ 101.

I often suggest that empirics be taught in 101. My idea is to teach some simple empirical methods alongside some simple theories, so that 101 students get an idea of how economists tell good theories from bad ones. The idea is not to remove theory from 101. Unfortunately, I seem to have convinced Allison that this is what I was suggesting, for she writes:
Noah Smith also thinks economics education needs an overhaul. He believes statistics, rather than theory, should be central to economics training. Right now statistics is important, but as a tool to help understand and validate theory. It is not taught until students are further along in their economics education. Smith thinks it should be the other way around in order to prevent students from putting too much weight on one theory. 
But for the same reasons the British students are wrong, Smith misses the point. Statistics without theory is pure data science. It doesn’t tell you much about why certain relationships, say inflation and unemployment, matter. Nor does it offer insight into how different economic factors influence each other. Statistics are a useful way to validate theories and infer which one is the right one to use. Data empower researchers to understand how and why different theories sometimes fall short. But without a solid theoretical grounding pure statistics lack meaning.
I agree with Allison that the case for teaching empirics with no theory in 101 is quite weak. This is just as true in econ as in physics or chemistry. Every intro science class includes some theory, and there's no reason econ should be any different. I don't see anyone making the case that theory should be absent from 101, and I certainly have never made such a case.

But what about theory with no empirics? That's the status quo in most econ 101 classes. Empirical results are sometimes cited, but empirical methods are almost never taught. This stands in stark contrast to physics, chemistry, and biology, where high school classes and even junior high classes have important lab components.

Allison doesn't really defend this status quo. She merely describes it, stating that empirical methods are "not taught until students are further along in their economics education." That is true, as things stand. But simply saying "This is the way we do things now" does not constitute an argument that this is the way we should continue to do things.

So what is the case for leaving empirical methods out of 101? Allison doesn't make one. The only cogent case I've seen so far, in fact, is the idea that empirical methods are just too hard for 101 students. Steve Williamson made this case in a comment a little while ago:
In theory, [teaching empirics in 101] seems fine, but the key constraint on what you can do in econ 101 is the capacity of the students. It would be great if we could teach econ 101 to students who are already tooled up in math and statistics. Also, it would be great if they would come to class, pay attention, and not go to sleep.
Steve seems to be overestimating the difficulty of empirical methods. A lot of the "credibility revolution" methods are really just a comparison of two means. How hard is that? It's a lot easier than using supply-and-demand graphs to analyze the deadweight loss from an import tariff. As for calculating statistical significance, if you think 101 students aren't mathematically sophisticated enough to understand it, just show them how to push a button and read a p-value from a table. That's no harder than showing high school chemistry students how to mix two liquids and saying "Look, it turns blue!".

Steve also has a pretty cynical view of the purpose of 101 itself:
[E]con 101 is often a general education requirement in undergrad programs, and it's high enrollment - helps to justify your existence to the rest of the university. It's meant to draw in students, and has been dumbed down to the point where the only tool that is taught is two curves that cross and shift around.
Call me a wild-eyed crusading idealist, but I think that if 101 is a general education requirement, it should be more than just a slick marketing tool designed to get more students (read: money) for the econ department. If it's a requirement, then we have a duty to teach more than just simple ideas - we should teach students how to evaluate those ideas in the context of reality.


Update

Some people have been asking me to explain how empirics would be integrated into 101, and I was about to try, but Steve showed up in the comments and explained it really well:
I can see where you're going though. So, suppose you integrate the empirical work with the theory. You need to structure the course so that you learn the methods as you go along. Here's some theory. Here's some data where you get a lot of information just from the scatterplot. Here's a theory that allows us to illustrate how to use regression. Here's something that illustrates omitted variables, etc. So, you build up the methods as you go along, and it's far more interesting than the specialized approach - do statistics in this department, do economics in this department. Students have no idea why they're learning stuff, and they just forget it. You could bring the math in too - much more interesting than just doing calculus for its own sake[.]
This is exactly how I was envisioning it. I'd start, though, with stuff even simpler than a linear model - just a randomized lottery experiment with a comparison of means. Like a school voucher experiment, for example.

Steve also makes a great point about what needs to be done politically for this to happen:
OK, now I'm selling myself on it. It's an uphill battle though. You have some behemoth institutions that are very conservative in your way - universities and publishers.
Yep, sounds about right. I think it will be crucial to either create an alternative to Mankiw, or to force Mankiw to do a major rewrite (which will of course be hard, since Mankiw himself is a third conservative behemoth institution).

Acemoglu, Laibson and List are already crafting an alternative. Their series of intro textbooks goes heavy on the empirics and has a whole software package, MyEconLab, designed to introduce kids to empirics in an easy yet useful way. This kind of software has the potential to become the econ equivalent of a high school chemistry lab. Econ educators, you might want to take a look at Acemoglu, Laibson, and List. It's pretty pricey, though.

Also, interestingly, it looks like INET is starting to emphasize empirics. That's good to see.

29 comments:

  1. But Economics is nothing like high school Physics or Chemistry in the slightest - it's a social science, we can't do simple controlled experiments in a classroom like you can with Chemistry or physics. I just don't understand the logic behind "it's done in those classes so we should do it too", why? It's really really hard to do good, robust, econometrics; it can barely be taught in a single econometrics course, teaching it as a portion of EC101 would barely teach them anything useful.

    I just have no idea what problem you're trying to solve. Economics students *will* get taught empirical methods in dedicated courses, so it doesn't help actual economics students. So basically you're trying to solve a theoretical problem where a hypothetical student will take only theoretical courses, including one EC101 course and courses from many other subjects, and might end up getting no empirical education. Why is this the Economics department's problem? Why would you sabotage the quality of what is meant to be an introductory literature/textbook review course and shoehorn half of another course in just to help this hypoethical student? Why can't instead the university itself force students of this type to take at least on dedicated empirical course.

    ReplyDelete
    Replies
    1. we can't do simple controlled experiments in a classroom like you can with Chemistry or physics

      You can use other people's data, as many economists themselves do.

      You can look at lottery studies, RCTs, and natural experiments, like economists do.

      It's a little different but it's not that different.

      Delete
    2. what is science ?
      it is a coherent way of thinking that emphasizes critical examination of data, and generating data based theories with predictive value
      much more important then any piece of knowledge, even basic stuff like newtons laws or that the atom and cell are the basis of chemistry or biology, is this idea of what science is.
      I think you could have good class experiments that inculcate this spirit of looking at data critically, and generating theories

      eg, does an increase in price really lower the amount sold ? how do we know this ? how general are our examples ? what are the counter examples ? how general is this ?

      Delete
  2. "Call me a wild-eyed crusading idealist..."

    Yes, you and Bernie have a lot in common. It's true that there is plenty of user-friendly software. A student can push a button and get a p-value, but it would be nice if they knew what it meant.

    I can see where you're going though. So, suppose you integrate the empirical work with the theory. You need to structure the course so that you learn the methods as you go along. Here's some theory. Here's some data where you get a lot of information just from the scatterplot. Here's a theory that allows us to illustrate how to use regression. Here's something that illustrates omitted variables, etc. So, you build up the methods as you go along, and it's far more interesting than the specialized approach - do statistics in this department, do economics in this department. Students have no idea why they're learning stuff, and they just forget it. You could bring the math in too - much more interesting than just doing calculus for its own sake (though I find algebra relaxing).

    OK, now I'm selling myself on it. It's an uphill battle though. You have some behemoth institutions that are very conservative in your way - universities and publishers. The latter are rapidly becoming dinosaurs though, so there's hope (don't tell my publisher I said that).

    ReplyDelete
    Replies
    1. You need to structure the course so that you learn the methods as you go along. Here's some theory. Here's some data where you get a lot of information just from the scatterplot. Here's a theory that allows us to illustrate how to use regression. Here's something that illustrates omitted variables, etc. So, you build up the methods as you go along

      YESSSSSSS, exactly!!

      Delete
    2. Do you know some rich people who could start a university?

      Delete
    3. Hmm, I'll try to meet some.

      Delete
    4. Sadly all the big hedge fund guys seem intent on giving money to existing universities, the tech bazillionaires seem intent on giving their money to help Africa and such, and the oil barons seem intent on using their money for pol'tics.

      Delete
  3. There aren't a lot of examples with data in Krugman and Wells's or Mankiw's intro books, and I agree there should be. One question though: which empirical results would you include?

    I could see e.g. Vernon Smith's experimental results for micro, but would there be any macro results that 1) test a theory that is at the undergraduate level 2) use stats methods at the undergraduate level?

    ReplyDelete
    Replies
    1. does the law of supply and demand really hold. How do we know this.
      counter examples

      Can we measure utility ? how do we measure the utility of a new ipod and good health on the same scale

      Delete
  4. Most large colleges have multiple sections of Econ 101, why not have a section for students with better math skills. Back in the day, when I was a student, I was taught Econ 101 without calculus. Did I learn the core concepts as well, probably better since I couldn't just treat problems as math problems

    ReplyDelete
  5. Anonymous8:53 PM

    As for statistics at the undergraduate level, why should statistics be a prerequisite for introductory economics? Here is how to deal with statistics for students who have not taken a course yet. Make use of the interocular traumatic method. Any statistical result that has a large enough effect will have a presentation, usually a graph, that hits the student right between the eyes. :)

    ReplyDelete
    Replies
    1. because small changes over time have a big effect: the diff to your retirement account of 1% or 2% over inflation is a lot over 30 years

      Delete
  6. Anonymous8:59 PM

    And don't forget simulation software. Netlogo is good, for instance.

    ReplyDelete
  7. I’ve tried to avoid the marketing shtick, but here it’s unavoidable. You may not like my intro macro textbook for lots of other reasons, but you should look at it just to see how I integrate empirics and theory. After the first third of the book, which is largely descriptive about the macroeconomic world, macro institutions and everyday macropolicy, the next section dives into a fairly standard simplistic Keynesian model. But it’s treated empirically! I show how a simple linear consumption function can be fitted to data and how out of sample data can be used to test it. In fact, a simple linear model of national income determination can be fitted in a series of steps the same way, something I always do as a lab exercise for the students. And every subsequent theoretical development is treated as an empirical proposition, with the simplest possible time series methods.

    I am totally with Noah on this: economics should be a lab science from the ground up. Teach elementary empirical methods along with elementary theory, not in parallel but together, each enriching the other.

    Incidentally, the politics of making economics a lab science are different from chemistry, biology etc. In the natural sciences most of the basic ideas have been nailed down empirically for decades or centuries, so students validate the theory by doing the empirics correctly. Not so economics. But that’s OK, since encouraging a critical, open-minded attitude is a good thing in an unsettled discipline.

    ReplyDelete
  8. This program makes sense for some micro problems: well defined outcome spaces, few driving factors, large data sets.

    However, it's bound to fail for macro. Unless economists themselves dump the complexity and focus on issues of parsimony and robustness. Little hope for that, though. You guys should teach some basics from machine learning and especially the distinction between bias and variance and the dangers of fit in small samples. They are relatively simple and should inoculate students against majority of macro research.

    Acemoglu stuff looks quite promising, but the heterodox one is utter BS: dynamics, institution? Say hello to (politically motivated) overfit.

    ReplyDelete
  9. But the very beginning of empiricism is not statistics or even experimentation, but careful, systematic observation, and public, repeatable, meaningful measurements. Start with that! If you don't think these need to be taught, look at this example. Or this one, which uses output per worker as a measure of productivity. That is completely inappropriate even within Europe, where hours worked per year vary widely (output per hour worked is a far better measure).

    ReplyDelete
  10. It's like today's Econ 101 students are eating a giant jar of peanut butter, and then only later eating jars of jelly in the 200/300/400/500-level courses... and all you're suggesting is that Econ 101 (and above) start feeding the kids peanut butter and jelly sandwiches.

    I think it sounds reasonable but without specifics I can see how it sounds quixotic to others.

    Econ 101 textbook authors are re-writing their textbooks all the time -- have you thought about giving specific suggestions about where and how exactly to slip in the jelly for the next edition? You could even go chapter by chapter or unit by unit or theory by theory, and suggest specific empirical methods, papers, tests, "lab" exercises, etc. to accompany the theory.

    ReplyDelete
  11. The point of any discipline is to accumulate knowledge, not recycle it. They way to do this is to test theory against results at every step, and constantly search for more results and better tools. This is not just "science" - any recent good book on, say, ancient Athens is a lot better than anything published 40 years ago, because it will integrate knowledge gained from archaeology, epigraphy, coin distributions, forensics, pollen analysis, different ways of textual criticism and much more. Statistics is one set of tools, but we lack good numbers for anything previous to 50 years ago (which in macro terms is last week - economies evolve slowly). So economics students need to be told of the limits of the evidence, how to gather and exploit proxy evidence, the limits of robustness, where tests between competing theories remain weak or out of reach. Just as history students are taught about primary and secondary sources, chains of evidence and the synergies (or clashes) between different sorts of evidence. In short, any first year economics course needs to inculcate a real sense of how very tentative most conclusions are, and the excitement of finding ways to strengthen them.

    ReplyDelete
  12. For better or worse academic social science is very stat heavy. This would include both the sort of experiments you might get in psychology (which probably merge into behavioural economics) and the observational studies you see in human geography. Other theory heavy and observational data heavy subjects such as ecology are also very stat heavy and do not see theory and empirical data study as distinct.

    ReplyDelete
  13. Not entirely sold on your plan yet. I found understanding the arguments (theory) before learning the details (empirics) quite useful. Taught me a way of thinking and approaching economic problems - and made the empirics part a lot more fascinating once I got to it. I reckon some HET and humility in the first econ Courses would be much beneficial, rather than the "the textbook says x" attitude of many professors.


    As for Coyle's article, I wish it was representative for the Student community; it just isn't... most students fuel a ideologically-driven hatred for the "neoliberal agenda" and want to replace the mainstream NNS power over economics with marxist and PKE instead. so much for "plurality".

    ReplyDelete
  14. I find the name "MyEconLab" a depressing (and unsuccessful) attempt to be trendy.

    ReplyDelete
  15. Charles2:49 PM

    As comparison, most introductory psychology books include a research methods chapter - usually chapter two - and a heavy emphasis is made on citing research throughout the book (thousands of citations are included in an introductory psychology textbook to reinforce the research behind the concepts). In addition, research methods and statistics are courses required of majors in both psychology and sociology, and are usually taught out of those departments specifically for their majors.

    The addition of sites like MobLab do provide more opportunity to bring experiments into the classroom, which may be easier to do once or twice a term than it would be to overhaul the entire course to emphasize methods/statistics in each lesson.

    ReplyDelete
  16. What about playing an MMORPG and seeing how the economy works inside the game?

    ReplyDelete
  17. Simon9:48 PM

    I come for the economics. But I stay for the Easter Eggs.

    ReplyDelete
  18. Ben S. Bernanke used statistics in a 2002 paper to determine whether Fed policy shocks had any affect on a host of real variables (including real GDP) in the USA from 1959 to 2001. Here is what he found: the Fed policy shocks had 3.2% to 13.2% out of 100% on any change on a variety of variables, using regression analysis. Google the FAVAR paper. This range of change is 'statistically significant' but common sense tells you it's not that big a deal. Next time a monetarist tells you how the Fed can change the world, throw this stat at them.

    ReplyDelete
  19. Gerard MacDonell7:31 PM

    I agree with your point about empiricism, but it is not well suggested by that photo of a high school chemistry lab. I have crystal clear memory that the main lesson taught in those lab sessions is how to fudge the data to fit the preconceived answer provided by theory. That theory has been tested elsewhere, but sure as hell not in that lab setting. Not to distract from your main point, which seems right. Like Brett mentioning his love of lamp, I just want to weigh in here with something.

    ReplyDelete
  20. Excelets are perhaps the tool you have sought
    http://academic.pgcc.edu/~ssinex/excelets/

    Excelets are interactive Excel spreadsheets or simulations of mathematical models. The user changes a variable and the spreadsheet changes in numerical, graphical, and/or even symbolic form (equations). Through the use of numerical experimentation and "what if" scenarios, we have a powerful learning tool for students using readily available off-the-shelf software. All of this is done computationally with no use of programming, no macros. An Excelet resembles a "Javaless" applet.

    Interaction is via sliders, radio buttons, menus, scroll bars, etc. as well as setting values in cells.

    ReplyDelete
  21. This comment has been removed by the author.

    ReplyDelete