The "let's lump a bunch of models together" approach

Volker Wieland and his coauthors have taken a first step toward addressing the problem of model uncertainty in macroeconomics. Basically, they have taken a large number of macro models and thrown them in a database, allowing you to compare across models or just to get the average of the models' predictions.

Now, of course, it is easily possible to point out a bunch of flaws in this approach. How will the set of models be chosen? How different are the models from each other? How would the inclusion of an utterly spurious model be detected and prevented? Is there publication bias at work? How can the database's users know what it means to give the models various weights? And so on. But in general, I think that this database is a good thing to do. Anything that makes model uncertainty explicit is good, and I think Wieland et. al. have executed this well.

However, I would like to point out specific error that I think Wieland et. al. may be committing here. In fact, Robert Waldmann has already pointed it out, in reference to the modeling approach used by the Bank of England. (Robert Waldmann does not write with a layperson audience in mind, so let me paraphrase instead of quote.) Basically, the Bank of England has what they call a "core" model, which is a microfounded DSGE model that contains only parameters that are assumed to be policy-invariant - in other words, a model Bob Lucas would approve of as "structural". Then it lumps that "core" model in with a bunch of other non-structural, ad-hoc type models to make a "hybrid" model. Then it uses the "hybrid" model for policy analysis.

Waldmann's point is this: Why bother with the "core" model in the first place? As soon as you lump it in with stuff that isn't "structural," it becomes pointless. If you're going to have a non-structural model, just use a non-structural model and don't bother with the DSGE thing.

The same problem applies to the database constructed by Wieland, et. al. It appears to contain a mix of microfounded, potentially "structural" DSGE models and non-structural ad-hoc models. But these two types of models don't mix.

As I always say, there are two things we might want from macro models: 1. forecasting, and 2. policy analysis. If all we want to do is forecast, we don't need to worry about whether our models are policy-invariant. In fact, making them policy-invariant will probably make them worse at forecasting, since it will severely shrink the number of variables in the model. So if forecasting is what we want, we don't need the structural models, and might as well toss them out, since they will probably just add noise to the consensus forecast.

But if policy analysis is what we want, then we need to answer the Lucas Critique. In this case, we don't want the ad-hoc models. We want only the policy-invariant models. The ad-hoc models should be tossed out. (Actually, in practice, it's hard or impossible to know if a parameter is really structural. But let's put this issue aside for now.)

So the full Wieland database is not the right mix of models for policy analysis, and it is probably not the right mix of models for forecasting either. To really use this database as it ought to be used, one should be able to easily toggle "only structural models".

Update: Waldmann has his own critique of the Wieland approach, which is different from (and partly contradicts!) mine.