The original question was: "Why has Japan performed poorly relative to 1990-vintage forecasts?" My answer was: "Because 1990-vintage forecasts were overestimating Japan's ability to exceed the West in productivity." Ryan's answer is, basically: "Because Japan's monetary policy has been too tight."
In macroeconomics, it is very hard to generate a good "counterfactual" - in other words, it's very hard to know what would have happened if some policy had been different. History only happens once, after all. So what we usually do is come up with "stories", and then look to see how the evidence, informed by our theories, fits with each story. It's not exactly science, but it's what we have to work with.
So let me see if I can characterize the stories Ryan and I are telling about the Japanese economy.
Ryan's story: "Japan is (or was) a country with better institutions and/or technology than the West. In the absence of a demand shortage, Japan would be considerably richer than the larger nations of Western Europe, as it was in 1990."
My story: "Japan is (and was) a country with about the same quality of institutions and productivity as the West. In the absence of a demand shortage, Japan would be doing better than it currently is, but not significantly better than the larger nations of Western Europe."
It's essentially impossible to quantitatively measure the quality of a country's institutions. Hence, to evaluate these two stories, let's look at the notion that an aggregate demand shortage has caused Japan's economy to underperform for 20 years.
Here is a picture of Japan's employment-to-population ratio:
As you can see, about 4% of Japan's population stopped working over the period of 1995-2003. This resulted in an unemployment rise of about 3 percentage points, with the remaining 1 percentage point presumably being some mix of "discouraged workers", grad students, housewives, and kids deciding to play instead of work. In any case, suppose that proper monetary policy had prevented any of this decline from occurring. Japan's labor force would then be about 7% bigger than it currently is. Assuming productivity would be the same then as now, that would raise Japan's per capita GDP (PPP) by 7% from its current level. This would account for about half the gap between Japan and Germany that Ryan shows opening up since 1995, and a quarter of the gap between Japan and the United States.
What would account for the rest? Two things: 1) total factor productivity, and 2) hours worked by the employed. Let's assume for the moment that TFP is determined entirely by institutions, and so let's look at how Japanese work hours per employed person have changed over time:
The steady drop in working hours since 1988 (most of which, interestingly, happened before 1995) corresponds to a reduction in the Japanese workweek from about 47 hours per week to just under 42. In comparison, the U.S. workweek is just under 39 hours, and the European workweek is just under 38.
Whose story does this support? It depends on what you think Japanese people's work habits would be if monetary policy had been looser. If the Bank of Japan had circumvented the zero lower bound by doing a bunch more quantitative easing, would Japanese people still be working 47 hours a week? Maybe, maybe not. A somewhat related question is whether that would be a good thing.
But note that the graph of hours worked provides a very natural explanation for the question of how Japan was managing to outperform the West before 1990. The answer: they worked more. A lot more. The only way they could have continued to outperform the West was by continuing to work more.
(Side note: OK, so what if TFP isn't exogenous? It's hard to think of why this would be the case. Labor hysteresis isn't an explanation, because for skill loss to matter, unemployed people have to come back to work, which hasn't happened in Japan (except slightly in 2003-2007). And endogenous growth isn't the answer, since Japanese total R&D spending has strongly outpaced the West since 1995. So I don't find it plausible that persistently tight monetary policy caused Japan's TFP to stagnate.)
So was what happened to Japan a preventable tragedy? I think part of it was a minor preventable tragedy. More aggressive easing probably could have saved up to 3% of Japan's working-age population (over 2 million people!) from unemployment. And in fact, Japan's bright spot in the mid-2000s came after the BOJ did a bunch of quantitative easing. So I think Ryan's story is right.
But I don't think it is the whole story. Japan stopped outperforming the West in large part because they started working less (though still more than people in the West). That change mostly happened well before the zero lower bound was hit, and began even before the Japanese asset bubble reached its peak. So I suspect not all of it could have been reversed by quantitative easing. But even if some could have been reversed, I do not see the end of the 47-hour workweek as a tragedy. Knowing what I know of Japanese society, I see it as a minor triumph.
Update: It has been brought to my attention that the "employment-population ratio" for Japan is not defined
Update 2: Matt Yglesias weighs in with a much better overview of the vagaries of PPP than I provided in my last post. But my argument applies perfectly well using PPP, which is why I didn't want to go there in this new post. Still, give Matt's post a read.
Update 3: Karl Smith weighs in as well. He points out that Japanese unemployment has mostly been youth unemployment, as Japanese companies are unwilling to fire older workers. He also shows that Japan's GDP per hour lags behind Italy's, something I did not know. That does not, however, mean that Japan's institutions and/or technology are worse than Italy's, however, since working more hours makes productivity/hour go way way down, as any grad student has experienced when finishing his or her dissertation. We should not expect GDP/hour to converge between countries whose work weeks are very different in length, even under the Solow model.