Saturday, December 03, 2011

Harrison & Kreps 1978: The power of irrational expectations

In the past few weeks I've had discussions with several different people about why financial markets are different from normal markets. I've come to realize that there is a very deep and fundamental fact about financial markets that almost nobody in the lay public - and a good chunk of people in the finance industry itself - don't understand. And that fact just happens to have implications that also shake the foundations of modern macroeconomics.

So it's time for another addition of Papers You Should Know. Today's paper is "Speculative Behavior in a Stock Market With Heterogeneous Expectations" by J. Michael Harrison and David M. Kreps (Quarterly Journal of Economics, 1978).

Before we talk about Harrison & Kreps, we need to understand why financial markets are so weird. In a nutshell, it's this: In normal markets, people who know exactly what they are trading will often still be willing to trade. In financial markets, people who know exactly what they are trading will almost never be willing to trade. For a quick explanation of why this is true, I turn to Brad DeLong:
In a standard economic transaction, it is no mystery where the value to both sides comes from. When I buy a double espresso from CafĂ© Nefeli for $2.25, the coffee is more valuabe to me then $2.25 is...The sources of the gains from trade are obvious. 
But in finance neither side is getting useful commodities. Instead, both sides are trading away claims to a pile of money and getting claims to a different pile of money in return. So how is it that me selling this pile of cash I have to you for that pile of cash that you currently own can be a good idea for both of us? Doesn't one of the piles have to be bigger? And isn't the person who trades the bigger for the smaller pile losing?
Think about that for a second. If I offer to sell you a share of stock for $10, why would you buy it? Well, you might buy it because you want to diversify or otherwise adjust your investment portfolio (to give you, say, more risk or longer maturity). But - let's be frank - chances are you'd buy it because you think that sometime in the future it'll be worth more than $10. Right? So now ask yourself: If I (the seller) also thought it would be worth more than $10 in the future, why the heck would I sell it to you for $10???

The answer is: I wouldn't. If someone offers to sell you a stock for $10 and tells you it's going to go up up up, in general don't buy it. Because if the guy selling you the stock really believed his own rosy prediction, he'd keep the stock for himself. The only time you should buy the stock is when you have good reason to believe that you are smarter or better-informed than the guy who's selling - in other words, if you think he's a sucker. Of course, he's only selling to you because he thinks you're a sucker.

So what we see is that while normal markets consist of people making trades because they have different preferences, financial markets mainly consist of a bunch of people with the same preferences all trying to sucker each other. This fact is called the "No-Trade Theorem," and economists have known about it for a long time. 

So in 1978, J. Michael Harrison and David M. Kreps decided to write down a model of a financial market in which everyone was trying to sucker everyone else. As far as I know, this was the first model of its kind. Even though few people believe that the model describes exactly how the real world works, it has become enormously influential, because it gives an idea of what the world would have to be like in order for us to observe the enormous volumes of financial trading that we actually see happening.

Briefly, the model works like this: Different people have different beliefs about the fundamental value of an asset (i.e., how much money the asset will pay them). One person thinks it'll pay A if the economy is good and B if the economy is bad. The other person thinks it'll pay C if the economy is good and D if the economy is bad. They know that they have different beliefs, they know what other people's beliefs are, and they "agree to disagree." So they are willing to trade; each one thinks the other one is a sucker.

So what price do they trade at? Well, you might think that the most bullish investor (i.e. the person who thinks it's worth the most) would just set the price. But actually, the price ends up being even higher than the most bullish person thinks it's worth! The reason is that the asset has resale value. You can buy it, collect some money from it, and then when the economy changes, you can sell it at a profit to someone who is more bullish than you are. So the price ends up being the sum of the asset's fundamental value and its resale value. Ta-da: Speculation!

Now, like I said, nobody really thinks this is exactly how things work. For one thing, people should learn over time - as the asset pays out money, people should update their beliefs. But Harrison & Kreps is not about describing the world, it's about exploring ideas. And the basic idea they explored has become one of the most powerful in all of financial economics. Since 1978, a number of authors have taken this basic notion of investor overconfidence and tried to either make it more realistic, or find it in the data. A few prime examples include Scheinkman & Xiong (2003), who put the overconfidence idea into a modern asset pricing model, Barber & Odean (2001), who find evidence that individual investors are overconfident and trade too much, and a number of "heterogeneous prior" models that allow people to learn as they go. All of these models owe something to the pioneering work of Harrison & Kreps.

But anyway, all that is preamble. This is actually a post about macroeconomics.

You see, the No-Trade Theorem says that financial markets shouldn't have a lot of trading. But we see a LOT of trading in these markets. And Harrison & Kreps showed that those trades are best explained by irrational expectations.

So what does that say about macro? Since the late 70s, nearly all of the models used by macroeconomists have been "rational expectations" models. "Rational expectations" is the idea that people don't make systematic mistakes when predicting the future. If you think that sounds a bit silly, you're not alone, but I kid you not when I say that rational expectations absolutely dominates modern macro.

But if expectations aren't rational in financial markets, why should they be rational in the economy as a whole? The answer is that they shouldn't. This is why Thomas Sargent, who won the Nobel Prize this year and who helped develop the theory of rational expectations, calls himself a "Harrison-Kreps Keynesian." Keynes, though he is usually associated with the idea of fiscal stimulus, was a professional stock speculator, and perceived clearly the irrationality of the markets in which he participated; Sargent is merely recognizing that financial market irrationality, which was formalized by Harrison and Kreps, is a huge hint that rational expectations is not going to get the job done in macro either. 

Pioneering macroeconomists like Sargent have spent a long time hacking through the wilderness of non-rational-expectations models. It is a daunting task, since there are infinitely many ways in which people could be irrational. Rational expectations lends itself to pure logical deduction - you can kind of just sit there and figure out how people should act. But to figure out how expectations really form, you need to get your hands dirty with things like lab experiments and careful empirical work.

But we really have no choice, if we want to understand the economy as it actually exists. As David Glasner says, "expectations are fundamental"; we can't afford to treat the process of human belief formation as an afterthought. That is the insight of Harrison and Kreps, and macro as a whole needs to take it to heart.

Update: Also see this article from today's NYT on Tom Sargent and Chris Sims (this year's other Nobelist). Both believe that modeling irrationality, as it exists in the real world, is the way to go.

Update 2: I just realized that a better title for this post would have been "Why is this market different from all other markets?" Ah, the "stairway wit"...


  1. I don't think rational expectations modeling requires the assumption that each individual agent has expectations that are genuinely rational, only that their deviations from rationality are random, so that they cancel out in aggregate.

  2. Hence the phrase "don't make systematic mistakes". ;)

  3. Anonymous5:05 PM

    As a financial adviser, I find one major flaw in your argument: heterogeneous time frames, or the lack thereof. Markowitz and Sharpe also leave mismatched time frames out of their capital market theories, but I find that a huge amount of "irrationality" is actually the work of investors' and traders' clashing of time frames.

    For example, when a trustee has a duty to raise cash to pay for a beneficiary's medical emergency, then the "sucker" price to sell a stock doesn't enter into it. The duty is simply to sell at the price offered in the market. To wait for a better price is, flatly, illegal.

    Such cases do present a tiny minority of day-to-day market action, but your theory would be better served dividing the market between traders (short-term) and investors (long-term) at least, as market action between investors does not need to fall back on irrational expectations.

  4. Anon, you're right, and I just lumped that into the category of "adjusting one's portfolio." I realize that there are heterogeneous preferences with regards to the heterogeneous characteristics of different financial assets. However, I just don't think that accounts for most of the volume of trade going on. And I think most fund managers and traders are out there looking for "alpha". I'd be VERY VERY skeptical of anyone who told me that that is not the case.

  5. Just look at NYSE volumes: Millions of shares per day in the 80's. Billions of shares per day today. Note that there are neither 1000 times more issues to trade, nor a thousand times greater population to be desiring these transactions.

    Anon's objection becomes less than irrelevant decimal dust in a billion share trade day.

    What Mr. Harless suggest about cancellation assumes no systematic biases, let alone mistakes.

    Noah says there are an infinite number of ways to be irrational. Possibly true, but is it relevant?

    The most likely irrational market action is herding behavior. This involves, by definition, a powerful systematic bias. It is also exactly as rational as a cliff full of lemmings.

    Consider also, the holding duration for the average transaction is on the order of 20 seconds. This is not a long-vs short time horizon phenomenon. This is programed trading mechanisms extracting minutes rents billions of times per day. It is rational behavior, but it totally ignores expectations beyond a time frame that's shorter than how long you can hold your breath.

    Any market model that is not informed by these facts is worse than useless.

    Also, macro is bunk, pure and simple. It is irrational to believe in rational expectations.


  6. Anon- Another sanity check for an account of finance trade volume is to listen to CNBC for a morning. Or in that case an insanity check, 90% of what you'll here is indeed guys arguing about chasing Alpha.

    The odd thing is that we do get some actual information out of these irrational finance markets, though much less than is sometimes claimed by EMH enthusiasts.

    Here's an interesting paper on non-financial prediction markets indicating that.

  7. andrew1:00 AM

    I remember Robert Solow said something to the effect of "it's too bad that oftentimes the social scientists who are out there empirically studying actual human behavior aren't very good at modeling, and the good modelers are often allergic to the messy work of empirical observation of human behavior."

  8. Andy's comment, while common, is wrong. Rational expectations requires that every individual agent know the correct probability distribution of every variable. This is completely clear from the definition in any textbook.

    I think the confusion comes from the fact that the statement is true in linear rational expectations models, which is where the field started.

  9. Anonymous4:43 AM

    If I (the seller) also thought it would be worth more than $10 in the future, why the heck would I sell it to you for $10???

    The answer is: I wouldn't.

    If the seller needs the 10 bucks today in cash (liquidity) he would sell at 10.
    Or if the seller has the information of another stock that costs 10 but has bigger prospects for the future he would sell the first to get the latter.

    Your logic as well as the one from Brad Delong is flawed in many ways.

  10. Adam C5:29 AM

    So, is it fair to say that this best describes behavior in the derivatives market? There it's not clear that anything has to do with shuffling risk or dealing with time frames as they pertain directly to the asset. I.e., it's a wager that is separate from the asset in question and thus really is based on one side thinking the other is a "sucker"?

    IANAE, but curious to understand how this interlocks with more "rational" markets...

  11. There is a lot of heterogeneity of preferences in financial transactions on at least two dimensions, risk for expected return, and liquidity. And the risk part has lots of dimensions, variance, skewness, probability of a total loss over n years, probability of a p% loss over q years,...

    So, there's lots of heterogeneity among financial assets and preferences over these varied assets that can account for a lot of trading even if there were zero asymmetric information. And of course, someone might easily knowingly trade an asset that they think will go up, if they don't think it will go up soon enough.

    Nonetheless, I agree that asymmetric information is humongous in this market, making it very different from most markets, and from a textbook classical market. I commented on this a couple of months ago at Mark's place:

    A key thing to realize is that making money on Wall Street is very different from making money producing things.

    If you make money growing strawberries, and then you sell them to a consumer, that's a pretty Pareto optimal transaction: The consumer gives you money in exchange for strawberries. The consumer gets more personal value from the strawberries than from the money or else he wouldn't make the exchange. The strawberry producer gets more personal value from the money than from the strawberries or else he wouldn't make the exchange. And with strawberries you can safely assume that both parties are pretty accurate in their calculations of value, as there's not that much asymmetric information about strawberries.

    But with financial transactions there is often tremendous asymmetric information. A layperson sells his stock in March of 2009 wrongly thinking it will only plunge further. A very expert finance person buys it knowing it's substantially underpriced given its expected earnings. This was not a Pareto optimal transaction at all. The layperson seller did not sell because the stock was worth less to him than the money. He only sold because he very mistakenly thought it was, and the savvy finance expert took advantage of his ignorance. Now, there was an external effect that was good for society in that that pushed the price closer to an accurate efficient one, helping allocate capital better. But in many cases any external effect is a lot less than the gain to the finance expert, and in the case of some financial transactions the external effect is actually a negative one, a huge negative one.

    The bottom line is that finance billionaires don't always create more wealth than they earn, sometimes far from it. Asymmetric information is huge here and so are externalities, often huge negative ones, not just positive.


  12. "But actually, the price ends up being even higher than the most bullish person thinks it's worth!"

    Which is why we need short sellers: so the market can form a consensus of all informed opinion. A good reason never to buy an asset you can't short sell.

  13. I really like these kinds of posts, keep 'em up. You just moved up to the top of my favorite econ blogs.

  14. Anonymous1:45 PM

    This post on Harrison and Kreps 1978 typifies the problem of applying the terms "rationality" and "irrationality" to any decision, financial or otherwise, that involves a judgment of value.

    As "Anonymous" #1 and #2 implies, every rational decision is a claim about the appropriateness of the means to achieving an end at a particular time. Every decision, in this regard, is heterogeneous with regard to the time frame considered or to the end sought, which may involve not solely maximum profit (or minimal loss) but also varying degrees of necessity (paying taxes, putting kids through college, supporting a more profitable option, operating out of a mechanical formula to rebalance a portfolio).

    There is never simply a single value in a single time-frame to consider. And any as-if model that simplifies to profit is irredeemably flawed. You can't have an alpha in a world in which other people are acting randomly. NS's assertion that the majority of trading is done in rational pursuit of alpha is unprovable, as it the notion that deviations from rationality cancel each other out.

    Anyone interested in a long history of asset values might benefit from reading this study of 350 years of housing prices in Amsterdam. It's not the stock market, but it is relevant to an understanding of economic value. Between 1628 and 1974, the real annualized value of housing along Herengracht canal rose only .5%. But this long-term concealed temporal fluctuations influenced by large historical events. If you invested with a 350-year horizon in 1628 you hadn't done very well by 1974. But a lot of short-termers undoubtedly made a killing -- or lost a fortune -- along the way.

  15. Michael Lewis mentioned this in 'Liar's Poker'.

  16. @Barry: Yes indeed!

  17. Anonymous6:47 AM

    Speculation can also occur and be quantitatively important in models where agents have rational expectations: