Experimental finance, like all experimental studies of complex systems, suffers from a problem of "external validity" - it is not clear that a few inexperienced undergrads trading $20 worth of made-up assets in a lab is anything like a real-world market with thousands of supercomputer-armed professionals trading trillions of actual assets over the course of years. This doesn't mean that finance experiments are useless for studying market outcomes, but it does put a limit on their use.
This problem, however, is much less severe if what you are studying is not market outcomes, but individual behavior. An individual is easy to stick in a lab. The person sitting in the lab is probably pretty similar to the person sitting in front of eTrade buying stocks at home. This weakens the "external validity" critique; it's a lot easier to claim that individual trader behavior in a lab is similar to individual trader behavior in the real world.
There has long been a strain of behavioral finance that studies the mistakes that individual investors make. The most famous example is the work of Terrance Odean, who has shown in a number of empirical studies that small investors (a.k.a. you and I) trade too much, make dumb choices, and eventually lose money. This may or may not result in the overall market being efficient (so keep your pants on, EMHers!), but it is certainly bad for the average American household, which often ends up throwing its money in a toilet. Figuring out exactly why people are throwing their money in a toilet is a major goal of finance research. There are a number of hypotheses - Odean has suggested overconfidence and the disposition effect. One idea in my job market paper was that people over-rely on prices as signals of fundamental value.
But anyway, these aren't just theoretical questions that people can bandy about for fun. We can actually stick people in a lab and find out! Thus, it made me quite happy last Friday when I got to see Terry Odean present a paper in which he does exactly that. In "Bubbling with Excitement: An Experiment" (Lin, Odean, and Andrade 2012), the authors show their lab subjects videos designed to evoke various emotions - excitement, fear, boredom. The subjects then trade in small groups, in the classic market setup of Vernon Smith. They found that excited groups of traders give rise to bigger bubbles and longer-lasting bubbles. Unlike most papers in this literature, Odean makes sure to use a large number of trading groups (48, compared to the typical 5-20), allowing him to statistically test his hypothesis (naturally, this was more expensive than the typical experiment).
This is an important result, because it opens up avenues for more research. We need to find out why excitement makes people overpay for assets! This means more experiments, of course, but that's how science progresses. Compared to particle physics, experimental finance is incredibly cheap, and has potentially enormous consequences for human welfare (and for your pocketbook).
Of course, even though the "external validity" problem is reduced when asking about individual behavior, it is not eliminated. Real-world traders, even if they lack experience and sophisticated tools, often have huge amounts of time at their disposal and are playing for big stakes. These are critiques that future experiments will partially address, and eventually the results of experiments will have to make testable predictions about large empirical datasets. But the future for this sort of work looks bright. I think that one thing we need to do is to move beyond the simple setup pioneered by Vernon Smith, and create laboratory markets that can capture a wider array of phenomena (a view shared by Jorg Oechssler of the University of Heidelberg). This is one thing that I plan to work on while at Stony Brook next year.