In the increasingly contentious world of pop economics, you - whether an educated layperson, an economist-in-training, or even a professional economist unused to the rough-and-tumble of the mediasphere - may find yourself in an argument with an economist. And when this happens, you should be prepared, because many of the arguments that may seem at first blush to be very powerful and devastating are, in fact, pretty weak tea.
For this reason, I have constructed a short list of (a few of) the fallacious arguments that you may encounter in your travels, along with suggested responses. Remember, forewarned is forearmed! I give you:
Noah's First Seven Principles For Arguing With Economists
Example: "You say Theory X is wrong...but don't you know that Theory X is supported by Nobel Prize winners A, B, and C, not to mention famous and distinguished professors D, E, F, G, and H?"
Suggested Retort: Loud, barking laughter.
Alternative Suggested Retort: "Richard Feynman said that 'Science is the belief in the ignorance of experts.' And you're not going to argue with HIM, are you?"
Reason You're Right: Credentials? Gimme a break. Nobody accepts received wisdom from sages these days. Show me the argument!
Principle #2: "All theories are wrong" is false.
Example: "Sure, Theory X fails to forecast any variable of interest or match important features of the data. But don't you know that all models are wrong? I mean, look at Newton's Laws...THOSE ended up turning out to be wrong, ha ha ha."
Suggested Retort: Empty an entire can of Silly String onto anyone who says this. (I carry Silly String expressly for this purpose.)
Alternative Suggested Retort: "Yeah, well, when your theory is anywhere near as useful as Newton's Laws, come back and see me, K?"
Reason You're Right: To say models are "wrong" is fatuous semantics; philosophically, models can only have degrees of predictive power within domains of validity. Newton's Laws are only "wrong" if you are studying something very small or moving very fast. For most everyday applications, Newton's Laws are very, very right.
Principle 3: "We have theories for that" is not good enough.
Example: "How can you say that macroeconomists have ignored Phenomenon X? We have theories in which X plays a role! Several, in fact!"
Suggested Retort: "Then how come no one was paying attention to those theories before Phenomenon X emerged and slapped us upside the head?"
Reason You're Right: Actually, there are two reasons. Reason 1 is that it is possible to make many models to describe any phenomenon, and thus there is no guarantee that Phenomenon X is correctly described by Theory Y rather than some other theory, unless there is good solid empirical evidence that Theory Y is right, in which case economists should be paying more a lot attention to Theory Y. Reason 2 is that if the profession doesn't have a good way to choose which theories to apply and when, then simply having a bunch of theories sitting around gathering dust is a little pointless.
Principle 4: Argument by accounting identity almost never works.
Example: "But your theory is wrong, because Y = C + I + G!"
Suggested Retort: "If my theory violates an accounting identity, wouldn't people have noticed that before? Wouldn't this fact be common knowledge?"
Reason You're Right: Accounting identities are mostly just definitions. Very rarely do definitions tell us anything useful about the behavior of variables in the real world. The only exception is when you have a very good understanding of the behavior of all but one of the variables in an accounting identity, in which case the accounting identity acts like a budget constraint. But that is a very rare situation indeed.
Principle 5: The Efficient Markets Hypothesis does not automatically render all models useless.
Example: "But if your model could predict financial crises, then people could use it to conduct a riskless arbitrage; therefore, by the EMH, your model cannot predict financial crises."
Suggested Retort: "By your logic, astrophysics can never predict when an asteroid is going to hit the Earth."
Reason You're Right: Conditional predictions are different than unconditional predictions. A macro model that is useful for making policy will not say "Tomorrow X will happen." It will say "Tomorrow X will happen unless you do something to stop it." If policy is taken to be exogenous to a model (a "shock"), then the EMH does not say anything about whether you can see an event coming and do something about it.
Principle 6: Models that only fit one piece of the data are not very good models.
Example: "Sure, this model doesn't fit facts A, B, and C, but it does fit fact D, and therefore it is a 'laboratory' that we can use to study the impact of changes in the factors that affect D."
Suggested Retort: "Nope!"
Reason You're Right: Suppose you make a different model to fit each phenomenon. Only if all your models don't interact will you be able to use each different model to study its own phenomenon. And this is highly unlikely to happen. Also, it's generally pretty easy to make a large number of different models that fit any one given fact, but very hard to make models that fit a whole bunch of facts at once. For these reasons, many philosophers of science claim that science theories should explain a whole bunch of phenomena in terms of some smaller, simpler subset of underlying phenomena. Or, in other words, wrong theories are wrong.
Principle 7: The message is not the messenger.
Example: "Well, that argument is being made by Person X, who is obviously just angry/a political hack/ignorant/not a real economist/a commie/stupid/corrupt."
Suggested Retort: "Well, now it's me making the argument! So what are you going to say about me?"
Reason You're Right: This should be fairly obvious, but people seem to forget it. Even angry hackish ignorant stupid communist corrupt non-economists can make good cogent correct arguments (or, at least, repeat them from some more reputable source!). Arguments should be argued on the merits. This is the converse of Principle 1.
There are, of course, a lot more principles than these, and I'll include some in a later post. The set of silly things that people can and will say to try to beat an interlocutor down is, well, very large. But I think these seven principles will guard you against much of the worst of the silliness. Keep them always with you at your side...Happy arguing!
34 comments:
Nice start. On the not-all-models-are-wrong point you write, "philosophically, models can only have degrees of predictive power within domains of validity"
Can you list a few models with degrees of predictive power which have been validated within limits? Give us your best shots.
A more specific and appropriate title might be "Seven principles for arguing with macroeconomists."
Surely Principles 1 and 7 are both examples of the same point? And as a respecter of both I would wish to suggest anonymously that there's no point arguing with economists. They spend all there time playing around with trying to pretend they are mathematicians without worrying about the real world.
@Anonymous 4:52: no. The points are related but not the same. Point 1 is that even geniuses are often wrong. Point 7 is that even morons are sometimes right; occasionally, they are even right for the right reason, which is what really matters.
Noah is being a little disingenuous in 7, though, because life is too short to pay attention to every crackpot - the world is full to overflowing with them. But if you are in a situation where you have to hear the argument, you might as well think about it.
... I'm not so sure. I'd put more emphasis on models being generally weak and having hard to quantify accuracy. Especially with respect to the future. ;-) I'd also frame this with the EMH being wrong, but wrong in a way that traders do not generally understand. Prices are not random, they are worse than that. ;-) We now know "not EMH" implies chaos.
Will have to chew on this, but it tastes good so far. Of course, there is an unstated premise — that it is possible to argue with economists. I can think of a few [cough, right-wing] economists with whom, to paraphrase Barney Frank, trying to have a conversation would be like trying to have a conversation with dining room table.
Regarding models, you can always retort with the complete version of Box's observation on model set theory:
“All models are wrong, some are useful” - George E.P. Box
See Box, George .E.P., "Robustness in the strategy of scientific model building," in Robustness in Statistics, R.L. Launer and G.N. Wilkinson, Editors. 1979, Academic Press: New York.
How about this:
When an economist says X must be true because a business person, politician or government says Y, the retort should be, "Let us discuss the probability that what the business person, politician or government says is actually true. Too often people do not question the truthiness of such people.
“All models are wrong, some are useful” - George E.P. Box
Perhaps, but Taleb isn't wrong in observing that we often learn where they fail in retrospect, and then call to a "do over."
re: "Robustness"
Isn't that the idea that while (or when) no single model has much validity, the sum of all models(provided they are independent in their assumptions) probably will?
This unfounded assumption was taken over from population genetics, if I am not mistaken, and is today used in climate science (see IPCC reports). It was originally characterized -- by a proponent! -- as "truth is at the intersection of all lies." Google it.
Correction: on "robustness" in model building Google
"THE STRATEGY OF" MODEL BUILDING by Levin"
"Modern economic analysis consists largely in building abstract mathematical models and
deriving familiar results from ever sparser modeling assumptions is considered as a
theoretical contribution. Why do economists spend so much time and effort in deriving
same old results from slightly different assumptions rather than trying to come up with
new and exciting hypotheses? We claim that this is because the process of refining
economic models is essentially a form of robustness analysis. The robustness of
modeling results with respect to particular modeling assumptions, parameter values or
initial conditions plays a crucial role for modeling in economics for two reasons. First,
economic models are difficult to subject to straightforward empirical tests for various
reasons. Second, the very nature of economic phenomena provides little hope of ever
making the modeling assumptions completely realistic. Robustness analysis is therefore a
natural methodological strategy for economists because economic models are based on
various idealizations and abstractions which make at least some of their assumptions
unrealistic (Wimsatt 1987; 1994a; 1994b; Mäki 2000; Weisberg 2006b)."
Swift would have a field day.
I hate to be so nihilistic, but I think economic "science" would benefit from a good dose of nihilism on the empirical side.
After all, it already embraces explicitly a form of moral nihilism: all else equal, we DON'T assume a dollar is worth more to a poor man than a rich one.
But then has there ever been a case of a decadent academic discipline reforming itself?
I generally distrust complex models, but then we do get to things like climate, where the alternative is to ignore. Not an ideal choice.
Well, Wikipedia's list of fallacies has grown a lot:
http://en.wikipedia.org/wiki/List_of_fallacies
But I wasn't able to find Verum ex quolibet. Maybe I didn't check well enough and it's under a different name, or they just thought that citing "Affirming the consequent" as a fallacy is enough given that Verum ex quolibet isn't really a fallacy from a logical point of view, but it becomes a problem when you're making science.
In particular with sciences based only on historical data I think it's easy, Verum ex quolibet could be quite appealing, we take a true assertion about a fact/event and then we elaborate a reasonable explanation of it, the whole argument seems true, isn't it?
Well, it often IS logically TRUE because if we say "P -> e" and e is a true assertion about a fact/event, P can be true or false, but "P -> e" as a whole will always be true. So where's the problem? When we have "P1 -> e" and "P2 -> e" and P1 and P2 are mutually exclusive so we cannot say just "(P1 AND P2) -> e" otherwise we get an Ex falso quodlibet, so, how do we chose between P1 and P2?
And that's where the need for P1 and P2 making falsifiable predictions about other singular facts/events kicks in. [by "other" I mean could be also a similar fact/event but happened elsewhere or "elsewhen"]
All this for saying that I agree with you on principle 3 but not too much where you say "unless there is good solid evidence that Theory Y is right". I would ask Theory Y to make also some testable prediction which can "go wrong" and which till now didn't go in that way.
This kind of prediction does not need to be about the future, we can make a theory explaining an event in 1985 and then go back in history looking if there were certain places with all those conditions required for an event similar to that of 1985 to happen. If nothing of that kind happened we have to understand why... if it did happen, well this doesn't prove our theory in any "positive" way but it makes it quite interesting of course :D
[I am just trying to restate K.Popper, nothing original from my side here]
Actually a more sophisticated retort to principal 5 is that the efficient markets hypothesis is generally only a statement about the mean of the distribution, not the distribution itself, because outcomes are probabilistic. Suppose (for example) there are two binary outcomes for a bank stock, A and B with probabilities p(A) and 1-p(A)=p(B). In state A, normality, the world is great and the stock returns r(A)>0. In state B (financial crisis) the bank stock is worth zero (return r(B)<0). Now, the EMH is only a statement about the the expected return prior to observing which state we are in. p(a)r(A) + (1-p(A))r(B) = risk free rate + risk premium. I can arbitrage away the expected return, but it does not imply p(B)=0. Even under EMH, forecasts are only right on average - they are a statement about the average of the distribution if one could observe the world many many times. Ex-post, we observe we are in state A or B - but it does not make the forecast wrong. In fact, since the forecast was the expected value, there was 0% chance it would ever come to pass.
"Even under EMH, forecasts are only right on average - they are a statement about the average of the distribution if one could observe the world many many times."
A cynic would call that an untestable assertion.
(Though, Mandelbrot did kill the normal distribution pretty completely.)
I've learnt some more principles:
1. One should avoid arguing with economists on economic issues where possible. The whole history of such discussions shows that economists are just getting stronger in their wrong opinions.
Any failure makes them stronger and they are very efficient in failures.
2. When discussion is inevitable, try to pull an economist to your professional ground, e.g. experimental physicists, and put some general scientific constraints. If
somebody has a success history and such a discussion has actually happened, please give details. I am not aware of such a lucky story.
3. Do not try to convince them with observations and models.
Since all "successful" economic models are based on autocorrelation (they call it autoregression also known as rational expectation ot EMH) you will never convince them with a model without AR.
Very nice piece. It reminded me of a theory I heard on a TED talk a while back explaining how people justify others holding conflicting beliefs than themselves.
The speaker explained that at first you assume the other person is ignorant, until they prove themselves knowledgeable, then you assume the person is stupid, until they prove themselves capable, at which point you assume the person is being purposefully misleading. I feel like this pretty well describes the current state of economic arguments.
I think 1 and 7 are just specific examples of the fallacies of an Appeal to Authority and Ad Hominem, respectively. They seem to be the most common and easily understood fallacies, so it is rather depressing that we keep seeing them so often.
john personna said...
" I generally distrust complex models, but then we do get to things like climate, where the alternative is to ignore. Not an ideal choice."
You do every day. You think that engineering models are simple?
As for climate science, the forecasts have been, if anything, too weak. The models can also hindcast, which allows testing (this is a big problem in economics, obviously).
On principle 2.
All models must contain an uncretainty term often missed in textbooks. In reality, physics is a big set of scatter plots described (interpolated, approximated or extrapolated) by a few equations. The gap between observations and predictions is the future research. No gap - no science.
Economic models are wrong in different sense - they are not aimed at reducing the uncertainty term by new and more accurate measurements. Macroeconomic time series (GDP, GDP deflator, unemployment, labor force, etc.) are not compatible over time. They have multiple unrecoverable breaks which make any quantitative analysis irrelevant.
Great post! I especially find #1 annoying and it is so common in the economics blogosphere ("but Keynes said...")
Barry, several points:
I don't actually use complex (the operative word) models every day.
I'm a non-fanboy semi-follower of Taleb, actually.
And while I might have some reservations about complex climate models, I do of course see co2 and temperature rising in the real world.
I think you have to be big believer in models to discount the AGW being measured, in an inversion of popular thought.
Could economists perhaps acknowledge this one little thing:
There is absolutely nothing in Economics which can tell us what policies we ought to adopt.
Any such attempt will be an "is-ought" fallacy. What policies we prefer depends on our individual preferences. Economics may inform those preferences and educate us on likely consequences of our actions. On rare occasions it might even be able to tell us what we ought to do assuming we wish to achieve a particular outcome.
But given that the desired outcomes are themselves typically matters of intense disagreement, Economics can never tell us what we “ought” to do.
In my observation, economists pay lip-service to this limitation. When pressed they will formally acknowledge the difference between positive and normative. But having done so, they go back to telling us how we ought to lead our lives: “You must accept what I say. I’m an economist!”
There are quite a few very good comments here by "anonymous". I find this annoying, as there is no way of knowing if the various "anonymous" comments are related.
I have an email-id I use for blogs which I do not use day to day. Such a strategy means that even fear of spam is no excuse.
Point 3 is too general to be correct, perhaps ever, surely in most important cases. Macroeconomists don't agree on much, so to say that "macroeconomists" missed any particular development is going to be untrue. Accusing a whole group of people who happen to have similar training of all thinking exactly alike is sloppy, and as sloppy thinking is wont to do, results in error. "A lot of macroeconomists missed X" is going to be true about most Xs, so why not go with that?
Reason, "anonymous" plus a time stamp should work, as in:
Anonymous@1:44AM, I disagree!!!
(I presume we are all seeing "host time" and not TZ adjusted.)
If this was a post about principles that would make the discussion with (or between) economist meaningfull - then I would agree one hundered percent.
However - I think most of these principles describe things that you would expect to hear from a economist rather than things they would call foul upon.
In real discussions with economists, there is one principle that rules them all.
The principle;
You are NOT allowed to call something simply wrong or stupid just beucase it is completely impossible to happen in the real world.
You are ONLY allowed to call something stupid or simply wrong if there is no way of creating a model, no matter how arcane, that would give this result as a consequence.
(OK - there is some rules about how you can construct the model - but that the domain of the model should, even roughly, correspond to the real world or that it,even roughly, should fit any real world facts, certainly is not two of them)
Your argument violates the XYZ hypothesis.
Rebuttal
Do you know the meaning of the word hypothesis?
Regarding your point number 4, I strongly agree. Are you surprised to hear that?
@Scott: I am not surprised.
I read your posts where you discuss the S=I identity, and I understand that you were not trying to argue that S=I implies that stimulus cannot work. You were using S=I to illustrate one part of a proof that the govt. spending multiplier does not depend on the degree of consumption smoothing (an argument that seems obviously correct to me). So Principle 4 does not apply to that discussion, as I see it.
hello.. this is my first time to visit your blog and read this posting, about the principles for arguing with economists. Ithink those interesting, but I wait for the others principles.
John Persona:
You objected to Barry, but you seem to have missed his point: he was pointing out that you *trust* complex models (not *use* them) everyday. (He's guessing that this occurs when you trust modern technology not to kill you at every turn.)
-Sean
===================
john personna said...
Barry, several points:
I don't actually use complex (the operative word) models every day. [snip]
===================
earlier:
Barry said...
john personna said...
" I generally distrust complex models, but then we do get to things like climate, where the alternative is to ignore. Not an ideal choice."
You do every day. You think that engineering models are simple?
[snip]
Post a Comment