In the increasingly contentious world of pop economics, you - whether an educated layperson, an economist-in-training, or even a professional economist unused to the rough-and-tumble of the mediasphere - may find yourself in an argument with an economist. And when this happens, you should be prepared, because many of the arguments that may seem at first blush to be very powerful and devastating are, in fact, pretty weak tea.
For this reason, I have constructed a short list of (a few of) the fallacious arguments that you may encounter in your travels, along with suggested responses. Remember, forewarned is forearmed! I give you:
Noah's First Seven Principles For Arguing With Economists
Example: "You say Theory X is wrong...but don't you know that Theory X is supported by Nobel Prize winners A, B, and C, not to mention famous and distinguished professors D, E, F, G, and H?"
Suggested Retort: Loud, barking laughter.
Alternative Suggested Retort: "Richard Feynman said that 'Science is the belief in the ignorance of experts.' And you're not going to argue with HIM, are you?"
Reason You're Right: Credentials? Gimme a break. Nobody accepts received wisdom from sages these days. Show me the argument!
Principle #2: "All theories are wrong" is false.
Example: "Sure, Theory X fails to forecast any variable of interest or match important features of the data. But don't you know that all models are wrong? I mean, look at Newton's Laws...THOSE ended up turning out to be wrong, ha ha ha."
Suggested Retort: Empty an entire can of Silly String onto anyone who says this. (I carry Silly String expressly for this purpose.)
Alternative Suggested Retort: "Yeah, well, when your theory is anywhere near as useful as Newton's Laws, come back and see me, K?"
Reason You're Right: To say models are "wrong" is fatuous semantics; philosophically, models can only have degrees of predictive power within domains of validity. Newton's Laws are only "wrong" if you are studying something very small or moving very fast. For most everyday applications, Newton's Laws are very, very right.
Principle 3: "We have theories for that" is not good enough.
Example: "How can you say that macroeconomists have ignored Phenomenon X? We have theories in which X plays a role! Several, in fact!"
Suggested Retort: "Then how come no one was paying attention to those theories before Phenomenon X emerged and slapped us upside the head?"
Reason You're Right: Actually, there are two reasons. Reason 1 is that it is possible to make many models to describe any phenomenon, and thus there is no guarantee that Phenomenon X is correctly described by Theory Y rather than some other theory, unless there is good solid empirical evidence that Theory Y is right, in which case economists should be paying more a lot attention to Theory Y. Reason 2 is that if the profession doesn't have a good way to choose which theories to apply and when, then simply having a bunch of theories sitting around gathering dust is a little pointless.
Principle 4: Argument by accounting identity almost never works.
Example: "But your theory is wrong, because Y = C + I + G!"
Suggested Retort: "If my theory violates an accounting identity, wouldn't people have noticed that before? Wouldn't this fact be common knowledge?"
Reason You're Right: Accounting identities are mostly just definitions. Very rarely do definitions tell us anything useful about the behavior of variables in the real world. The only exception is when you have a very good understanding of the behavior of all but one of the variables in an accounting identity, in which case the accounting identity acts like a budget constraint. But that is a very rare situation indeed.
Principle 5: The Efficient Markets Hypothesis does not automatically render all models useless.
Example: "But if your model could predict financial crises, then people could use it to conduct a riskless arbitrage; therefore, by the EMH, your model cannot predict financial crises."
Suggested Retort: "By your logic, astrophysics can never predict when an asteroid is going to hit the Earth."
Reason You're Right: Conditional predictions are different than unconditional predictions. A macro model that is useful for making policy will not say "Tomorrow X will happen." It will say "Tomorrow X will happen unless you do something to stop it." If policy is taken to be exogenous to a model (a "shock"), then the EMH does not say anything about whether you can see an event coming and do something about it.
Principle 6: Models that only fit one piece of the data are not very good models.
Example: "Sure, this model doesn't fit facts A, B, and C, but it does fit fact D, and therefore it is a 'laboratory' that we can use to study the impact of changes in the factors that affect D."
Suggested Retort: "Nope!"
Reason You're Right: Suppose you make a different model to fit each phenomenon. Only if all your models don't interact will you be able to use each different model to study its own phenomenon. And this is highly unlikely to happen. Also, it's generally pretty easy to make a large number of different models that fit any one given fact, but very hard to make models that fit a whole bunch of facts at once. For these reasons, many philosophers of science claim that science theories should explain a whole bunch of phenomena in terms of some smaller, simpler subset of underlying phenomena. Or, in other words, wrong theories are wrong.
Principle 7: The message is not the messenger.
Example: "Well, that argument is being made by Person X, who is obviously just angry/a political hack/ignorant/not a real economist/a commie/stupid/corrupt."
Suggested Retort: "Well, now it's me making the argument! So what are you going to say about me?"
Reason You're Right: This should be fairly obvious, but people seem to forget it. Even angry hackish ignorant stupid communist corrupt non-economists can make good cogent correct arguments (or, at least, repeat them from some more reputable source!). Arguments should be argued on the merits. This is the converse of Principle 1.
There are, of course, a lot more principles than these, and I'll include some in a later post. The set of silly things that people can and will say to try to beat an interlocutor down is, well, very large. But I think these seven principles will guard you against much of the worst of the silliness. Keep them always with you at your side...Happy arguing!
Nice start. On the not-all-models-are-wrong point you write, "philosophically, models can only have degrees of predictive power within domains of validity"
ReplyDeleteCan you list a few models with degrees of predictive power which have been validated within limits? Give us your best shots.
A more specific and appropriate title might be "Seven principles for arguing with macroeconomists."
ReplyDeleteSurely Principles 1 and 7 are both examples of the same point? And as a respecter of both I would wish to suggest anonymously that there's no point arguing with economists. They spend all there time playing around with trying to pretend they are mathematicians without worrying about the real world.
ReplyDelete@Anonymous 4:52: no. The points are related but not the same. Point 1 is that even geniuses are often wrong. Point 7 is that even morons are sometimes right; occasionally, they are even right for the right reason, which is what really matters.
ReplyDeleteNoah is being a little disingenuous in 7, though, because life is too short to pay attention to every crackpot - the world is full to overflowing with them. But if you are in a situation where you have to hear the argument, you might as well think about it.
... I'm not so sure. I'd put more emphasis on models being generally weak and having hard to quantify accuracy. Especially with respect to the future. ;-) I'd also frame this with the EMH being wrong, but wrong in a way that traders do not generally understand. Prices are not random, they are worse than that. ;-) We now know "not EMH" implies chaos.
ReplyDeleteWill have to chew on this, but it tastes good so far. Of course, there is an unstated premise — that it is possible to argue with economists. I can think of a few [cough, right-wing] economists with whom, to paraphrase Barney Frank, trying to have a conversation would be like trying to have a conversation with dining room table.
ReplyDeleteRegarding models, you can always retort with the complete version of Box's observation on model set theory:
ReplyDelete“All models are wrong, some are useful” - George E.P. Box
See Box, George .E.P., "Robustness in the strategy of scientific model building," in Robustness in Statistics, R.L. Launer and G.N. Wilkinson, Editors. 1979, Academic Press: New York.
How about this:
ReplyDeleteWhen an economist says X must be true because a business person, politician or government says Y, the retort should be, "Let us discuss the probability that what the business person, politician or government says is actually true. Too often people do not question the truthiness of such people.
“All models are wrong, some are useful” - George E.P. Box
ReplyDeletePerhaps, but Taleb isn't wrong in observing that we often learn where they fail in retrospect, and then call to a "do over."
re: "Robustness"
ReplyDeleteIsn't that the idea that while (or when) no single model has much validity, the sum of all models(provided they are independent in their assumptions) probably will?
This unfounded assumption was taken over from population genetics, if I am not mistaken, and is today used in climate science (see IPCC reports). It was originally characterized -- by a proponent! -- as "truth is at the intersection of all lies." Google it.
Correction: on "robustness" in model building Google
ReplyDelete"THE STRATEGY OF" MODEL BUILDING by Levin"
"Modern economic analysis consists largely in building abstract mathematical models and
ReplyDeletederiving familiar results from ever sparser modeling assumptions is considered as a
theoretical contribution. Why do economists spend so much time and effort in deriving
same old results from slightly different assumptions rather than trying to come up with
new and exciting hypotheses? We claim that this is because the process of refining
economic models is essentially a form of robustness analysis. The robustness of
modeling results with respect to particular modeling assumptions, parameter values or
initial conditions plays a crucial role for modeling in economics for two reasons. First,
economic models are difficult to subject to straightforward empirical tests for various
reasons. Second, the very nature of economic phenomena provides little hope of ever
making the modeling assumptions completely realistic. Robustness analysis is therefore a
natural methodological strategy for economists because economic models are based on
various idealizations and abstractions which make at least some of their assumptions
unrealistic (Wimsatt 1987; 1994a; 1994b; Mäki 2000; Weisberg 2006b)."
Swift would have a field day.
I hate to be so nihilistic, but I think economic "science" would benefit from a good dose of nihilism on the empirical side.
After all, it already embraces explicitly a form of moral nihilism: all else equal, we DON'T assume a dollar is worth more to a poor man than a rich one.
But then has there ever been a case of a decadent academic discipline reforming itself?
I generally distrust complex models, but then we do get to things like climate, where the alternative is to ignore. Not an ideal choice.
ReplyDeleteClimate models are not reflexive which is why a complicated climate model stands a better chance of being useful than a complicated economic model. The behavior of drivers *even in a tunnel* is impossible to predict but the behavior of oil flowing in a pipe is tractable.
Delete--bks
Well, Wikipedia's list of fallacies has grown a lot:
ReplyDeletehttp://en.wikipedia.org/wiki/List_of_fallacies
But I wasn't able to find Verum ex quolibet. Maybe I didn't check well enough and it's under a different name, or they just thought that citing "Affirming the consequent" as a fallacy is enough given that Verum ex quolibet isn't really a fallacy from a logical point of view, but it becomes a problem when you're making science.
In particular with sciences based only on historical data I think it's easy, Verum ex quolibet could be quite appealing, we take a true assertion about a fact/event and then we elaborate a reasonable explanation of it, the whole argument seems true, isn't it?
Well, it often IS logically TRUE because if we say "P -> e" and e is a true assertion about a fact/event, P can be true or false, but "P -> e" as a whole will always be true. So where's the problem? When we have "P1 -> e" and "P2 -> e" and P1 and P2 are mutually exclusive so we cannot say just "(P1 AND P2) -> e" otherwise we get an Ex falso quodlibet, so, how do we chose between P1 and P2?
And that's where the need for P1 and P2 making falsifiable predictions about other singular facts/events kicks in. [by "other" I mean could be also a similar fact/event but happened elsewhere or "elsewhen"]
All this for saying that I agree with you on principle 3 but not too much where you say "unless there is good solid evidence that Theory Y is right". I would ask Theory Y to make also some testable prediction which can "go wrong" and which till now didn't go in that way.
This kind of prediction does not need to be about the future, we can make a theory explaining an event in 1985 and then go back in history looking if there were certain places with all those conditions required for an event similar to that of 1985 to happen. If nothing of that kind happened we have to understand why... if it did happen, well this doesn't prove our theory in any "positive" way but it makes it quite interesting of course :D
[I am just trying to restate K.Popper, nothing original from my side here]
Actually a more sophisticated retort to principal 5 is that the efficient markets hypothesis is generally only a statement about the mean of the distribution, not the distribution itself, because outcomes are probabilistic. Suppose (for example) there are two binary outcomes for a bank stock, A and B with probabilities p(A) and 1-p(A)=p(B). In state A, normality, the world is great and the stock returns r(A)>0. In state B (financial crisis) the bank stock is worth zero (return r(B)<0). Now, the EMH is only a statement about the the expected return prior to observing which state we are in. p(a)r(A) + (1-p(A))r(B) = risk free rate + risk premium. I can arbitrage away the expected return, but it does not imply p(B)=0. Even under EMH, forecasts are only right on average - they are a statement about the average of the distribution if one could observe the world many many times. Ex-post, we observe we are in state A or B - but it does not make the forecast wrong. In fact, since the forecast was the expected value, there was 0% chance it would ever come to pass.
ReplyDelete"Even under EMH, forecasts are only right on average - they are a statement about the average of the distribution if one could observe the world many many times."
ReplyDeleteA cynic would call that an untestable assertion.
(Though, Mandelbrot did kill the normal distribution pretty completely.)
ReplyDeleteI've learnt some more principles:
ReplyDelete1. One should avoid arguing with economists on economic issues where possible. The whole history of such discussions shows that economists are just getting stronger in their wrong opinions.
Any failure makes them stronger and they are very efficient in failures.
2. When discussion is inevitable, try to pull an economist to your professional ground, e.g. experimental physicists, and put some general scientific constraints. If
somebody has a success history and such a discussion has actually happened, please give details. I am not aware of such a lucky story.
3. Do not try to convince them with observations and models.
Since all "successful" economic models are based on autocorrelation (they call it autoregression also known as rational expectation ot EMH) you will never convince them with a model without AR.
Very nice piece. It reminded me of a theory I heard on a TED talk a while back explaining how people justify others holding conflicting beliefs than themselves.
ReplyDeleteThe speaker explained that at first you assume the other person is ignorant, until they prove themselves knowledgeable, then you assume the person is stupid, until they prove themselves capable, at which point you assume the person is being purposefully misleading. I feel like this pretty well describes the current state of economic arguments.
I think 1 and 7 are just specific examples of the fallacies of an Appeal to Authority and Ad Hominem, respectively. They seem to be the most common and easily understood fallacies, so it is rather depressing that we keep seeing them so often.
ReplyDeletejohn personna said...
ReplyDelete" I generally distrust complex models, but then we do get to things like climate, where the alternative is to ignore. Not an ideal choice."
You do every day. You think that engineering models are simple?
As for climate science, the forecasts have been, if anything, too weak. The models can also hindcast, which allows testing (this is a big problem in economics, obviously).
On principle 2.
ReplyDeleteAll models must contain an uncretainty term often missed in textbooks. In reality, physics is a big set of scatter plots described (interpolated, approximated or extrapolated) by a few equations. The gap between observations and predictions is the future research. No gap - no science.
Economic models are wrong in different sense - they are not aimed at reducing the uncertainty term by new and more accurate measurements. Macroeconomic time series (GDP, GDP deflator, unemployment, labor force, etc.) are not compatible over time. They have multiple unrecoverable breaks which make any quantitative analysis irrelevant.
Great post! I especially find #1 annoying and it is so common in the economics blogosphere ("but Keynes said...")
ReplyDeleteBarry, several points:
ReplyDeleteI don't actually use complex (the operative word) models every day.
I'm a non-fanboy semi-follower of Taleb, actually.
And while I might have some reservations about complex climate models, I do of course see co2 and temperature rising in the real world.
I think you have to be big believer in models to discount the AGW being measured, in an inversion of popular thought.
Could economists perhaps acknowledge this one little thing:
ReplyDeleteThere is absolutely nothing in Economics which can tell us what policies we ought to adopt.
Any such attempt will be an "is-ought" fallacy. What policies we prefer depends on our individual preferences. Economics may inform those preferences and educate us on likely consequences of our actions. On rare occasions it might even be able to tell us what we ought to do assuming we wish to achieve a particular outcome.
But given that the desired outcomes are themselves typically matters of intense disagreement, Economics can never tell us what we “ought” to do.
In my observation, economists pay lip-service to this limitation. When pressed they will formally acknowledge the difference between positive and normative. But having done so, they go back to telling us how we ought to lead our lives: “You must accept what I say. I’m an economist!”
There are quite a few very good comments here by "anonymous". I find this annoying, as there is no way of knowing if the various "anonymous" comments are related.
ReplyDeleteI have an email-id I use for blogs which I do not use day to day. Such a strategy means that even fear of spam is no excuse.
Point 3 is too general to be correct, perhaps ever, surely in most important cases. Macroeconomists don't agree on much, so to say that "macroeconomists" missed any particular development is going to be untrue. Accusing a whole group of people who happen to have similar training of all thinking exactly alike is sloppy, and as sloppy thinking is wont to do, results in error. "A lot of macroeconomists missed X" is going to be true about most Xs, so why not go with that?
ReplyDeleteReason, "anonymous" plus a time stamp should work, as in:
ReplyDeleteAnonymous@1:44AM, I disagree!!!
(I presume we are all seeing "host time" and not TZ adjusted.)
If this was a post about principles that would make the discussion with (or between) economist meaningfull - then I would agree one hundered percent.
ReplyDeleteHowever - I think most of these principles describe things that you would expect to hear from a economist rather than things they would call foul upon.
In real discussions with economists, there is one principle that rules them all.
The principle;
You are NOT allowed to call something simply wrong or stupid just beucase it is completely impossible to happen in the real world.
You are ONLY allowed to call something stupid or simply wrong if there is no way of creating a model, no matter how arcane, that would give this result as a consequence.
(OK - there is some rules about how you can construct the model - but that the domain of the model should, even roughly, correspond to the real world or that it,even roughly, should fit any real world facts, certainly is not two of them)
Your argument violates the XYZ hypothesis.
ReplyDeleteRebuttal
Do you know the meaning of the word hypothesis?
Regarding your point number 4, I strongly agree. Are you surprised to hear that?
ReplyDeleteI am !
Delete@Scott: I am not surprised.
ReplyDeleteI read your posts where you discuss the S=I identity, and I understand that you were not trying to argue that S=I implies that stimulus cannot work. You were using S=I to illustrate one part of a proof that the govt. spending multiplier does not depend on the degree of consumption smoothing (an argument that seems obviously correct to me). So Principle 4 does not apply to that discussion, as I see it.
hello.. this is my first time to visit your blog and read this posting, about the principles for arguing with economists. Ithink those interesting, but I wait for the others principles.
ReplyDeleteJohn Persona:
ReplyDeleteYou objected to Barry, but you seem to have missed his point: he was pointing out that you *trust* complex models (not *use* them) everyday. (He's guessing that this occurs when you trust modern technology not to kill you at every turn.)
-Sean
===================
john personna said...
Barry, several points:
I don't actually use complex (the operative word) models every day. [snip]
===================
earlier:
Barry said...
john personna said...
" I generally distrust complex models, but then we do get to things like climate, where the alternative is to ignore. Not an ideal choice."
You do every day. You think that engineering models are simple?
[snip]
I'm sorry if my comments repeat things already noted in the comments thread. I arrived late. And life is too short to sift through everything that's been said. I find this post to be pretty simple-minded. But it also is typical of the way economists (especially young ones) proceed in the world. So let me propose a couple of rules to keep in mind when discussing economics.
ReplyDeleteHere is one: Economics is not physics. So the analogy you make between models in the two fields (consistently, I'd note) actually requires some argument. You provide none. Asserting analogiies to some other more 'scientific' enterprise is simply another way of invoking (credentials) authority. Show me the argument?
Here is yet another: Economic models are not (necessarily) predictive. Cases in point? Akerloff on the market lemons makes no prediction. Schelling on residential segregation makes no prediction. Your claim that empirical prediction is the (sole?) criterion for assessing models requires an argument. You provide none. Are Akerloff and Schelling (both Nobel Laureates to invoke authority in a wholly appropriate way - I take their work as exemplary, which is a good rule of thumb.) not scientific? If not, that is a costly exclusion; if so, how do their models fit your general (un-argued) view?
(I actually think economic models are not predictive all. But that is another matter. Let's just say lack of empirical predictions in no way entails that models are not central to scientific inquiry in economics.)
I suspect that you'll try to introduce some distinction between sorts of models - micro-, macro- econometric. Knock yourself out. But doing so means that you need to police discussions with economists to make sure they are not doing a two-step of the "Oh! I meant this other kind of model over here." sort. There is nothing in the discussion above that discriminates between types of models or their uses. So, a bit of clarity is in order. And, of course, so is an argument for any categorization you'd like to propose.
I know these are hopelessly philosophical matters. Too bad. Among the irritating aspects of arguing with economists is the fatuous way they assume that their tacit philosophical premises need no defense. It is not that, being an empiricist, you make no philosophical assumptions. Typically it simply means that you try ignore them and get away with invoking the authority of being an "ECONOMIST." Sorry, I'm unimpressed.
Asserting analogiies to some other more 'scientific' enterprise is simply another way of invoking (credentials) authority. Show me the argument?
ReplyDeleteAnalogies like that are for illustrative purposes only. No one expects economics to work like physics.
Economic models are not (necessarily) predictive.
How can we evaluate a model that makes no predictions? Acclamation and consensus? How good it intuitively feels? But that's religion, not science. If I'm going to take things on faith, I'd rather it be something interesting and fun instead of something difficult.
Noah,
ReplyDelete(1) Why are ALL of your illustrative examples from the sexier "real" science of physics? (OK - maybe not always, but pretty damned regularly.) Why not analogies to literature? Or history? You roll those out in order to differentiate your model-making from less reputable intellectual undertakings. And you use physics to gain credibility. Sort of like invoking Nobel winners as authorities. Actually not sort of - just like.
(2) You seem to think that the only criterion of assessment for scientific practice and progress is empirical. Hence (in your reply) you attempt to relegate anything NOT predictive to the sphere of religion.
When, say, Rubinstein argues that game theoretic models are incapable of predicting behavior, do you take him to be signing up for the seminary? Is he saying models provide no knowledge, they only shore up faith? Note, his is a strong claim - it is not that game theoretic models can, but often fail to, predict behavior, but that they are incapable of doing so.
Actually the history and philosophy of science suggests that there are multiple dimensions on which science is and should be assessed. See Phil Kitcher and/or Larry Laudan. These (minimally) are conceptual, technical, and yes empirical dimensions. If Schelling and Akerlof make no predictions are their models simply religion? They didn't win the prize for prophesy or theology. When you read the market for lemons and think it is a cool model, and also realize it makes no predictions, do you ask why is it cool? Or do you simply chalk things up to faith?
So we might think of models as devices for allowing us to talk about unobservable aspects of the real world (which include most sorts of causal mechanisms). In order to explain things we need to be able to identify mechanisms and examine the conditions under which they operate. Neither task involves prediction. Both are central to science in any description. But that would require you to dispense with the empiricist philosophical baggage. Prediction is over rated. Ask the folks who study plate tectonics or evolutionary biology.
Why are ALL of your illustrative examples from the sexier "real" science of physics?
ReplyDeleteBecause that's the only other subject I've studied. I don't really know much about literature or history.
You seem to think that the only criterion of assessment for scientific practice and progress is empirical. Hence (in your reply) you attempt to relegate anything NOT predictive to the sphere of religion.
It's "religion" only if you use models to make predictions or policy recommendations without empirical support.
When, say, Rubinstein argues that game theoretic models are incapable of predicting behavior, do you take him to be signing up for the seminary?
No, he is doing math. Math often does not predict anything real. that doesn't mean it's not worth doing. A professor of mine used to say "Math is the study of all possible worlds; science is the study of the world in which we happen to live." I agree with that.
They didn't win the prize for prophesy or theology.
Prize? "The prize" is just a bunch of guys getting together and deciding they like something. It means nothing. I actually think it's dumb.
In order to explain things we need to be able to identify mechanisms and examine the conditions under which they operate. Neither task involves prediction.
I think the last sentence here is very wrong. How do you "identify" a mechanism? Only by predicting how it will operate (under certain conditions) and then watching to see that it does operate that way.
I find the last point interesting, due to the fact that most mainstream economists level more and stronger arguments against the Right and most notably classical liberals than the lefty communists. Actually most macro-economists of "high" standing do actually have a leftist political stand. So I found your last comment rather curious.
ReplyDelete""Argument by accounting identity almost never works: But your theory is wrong, because Y = C + I + G!""
ReplyDeleteI think you're flipping it here (denying the antecedent, or whatever it's called). Obviously if a theory violates an accounting identity then it IS wrong (Paul Krugman in one of his books describes an argument with some politician who insisted that countries must run both a current account and financial account surplus simultaneously).
But there really aren't that many theories like that. What you do see is people making the opposite claim: my theory *obeys* the accounting identity therefore it must be true!
I cautiously disagree with Principle #1.
ReplyDeleteOf course, the substance of the argument matters more than reputation.
But I think the world would be better off if the general public actually lent MORE credence to highly authoritative economists rather than people like Niall Ferguson, John Stossle, some members of CNBC's staff, etc.
Most of the public doesn't have time to dig into the substance of the arguments.
They need signals to tell who to listen to.
And I suspect the problem runs contrary to your suggestion; the problem is that people don't care ENOUGH about actual, real credentials. If they did, they would pay attention to what Joe Stiglitz, Paul Krugman, Daron Acemoglu, Esther Duflo, and so on have to say.
Which, I think, would be a better outcome than the current equilibrium.