Wednesday, July 25, 2012

Judgment calls and philosophy of science


Lately I've done several philosophy-of-science posts on macro, complaining about what I call "judgment calls" (see here and here). I've been getting a lot of comments about philosophy of science, and I thought I'd take some time to step back and lay out how I think about the subject. So here's a post on Noah's General Philosophy of Science.

Preliminaries:

1. Language. I don't believe in demanding that terms be precisely defined, or in demanding that definitions be perfectly consistent. Reason: Defining terms precisely is incredibly hard. I find that philosophical conversations that focus too much on definitions and jargon quickly get lost in the weeds of "What do you really mean by X?". Sometimes you actually do need to drill down and figure out exactly how you're using a term, or how your usage is different from somebody else's, or how the usage depends on the situation. But I think that sort of intense focus on terminology should only be used sparingly, in times of great need.

2. Past Philosophers. I only care a little bit about what Popper, or Kuhn, or Feyerabend said. Not zero, because those guys were smart, and they read a lot of history and a lot of other people's ideas. But I only care a little, because I really want to figure these things out for myself instead of taking the word of an "expert", and I believe that I am intellectually equipped to do so (note: I am not trying to push the boundaries of philosophy-of-science scholarship here). So if I say something that conflicts with what Kuhn thought...well, as the Japanese say, sho ga nai ne! So to anyone who reads this and says "You are such an ignorant amateur, why don't you go read what some real experts say?!", I preemptively reply: "Why don't you try thinking for yourself instead of parroting an authority figure?" I see absolutely no reason not to reinvent the wheel occasionally. (If you're interested, my main sources of ideas were probably Charles Marcus, Robert Laughlin, Richard Feynman, Lee Smolin, Robert Waldmann, and Steven Smith, as well as some of those well-known philosophers. But some parts I think I just made up.)

3. Ontology. Ontology is the philosophy of what existence means. My ontology is basically what I think of as "pragmatist"...we believe things because it's useful to believe them. If you disbelieve in the existence of a wall, you're going to stub your toe and it's going to be unpleasant. Or maybe not...try disbelieving in the wall and let me know the result. I'll be over here with a beer, getting your experiment on video. That's basically my philosophy of what "existence" means. One result of this outlook is that I think of "detectability" (or "observability") is the same thing as "existence"...if you can't in some way, however indirectly, stub your toe on something, it might as well not exist. I don't know if that's what other people mean when they say "pragmatism", but see Point 1 about language.

4. Epistemology. Epistemology is the philosophy of how you can know things. When it comes to science, there are limitations on how much you can know. Tomorrow, things might all start falling up. You don't know that they won't! This is, I believe, called the "paradox of induction". "Laws" of the Universe might change tomorrow. If we're lucky, they won't change. So far, things still fall down. Whew! Also, tomorrow there might cease to be any sort of "laws" at all. See here and here for ideas about what might happen in that case. Suffice to say that it would be a weird, weird day.

OK, now that those are out of the way, on to My General Philosophy of Science:

5. The Goal of Science. The main goal of science, as I see it, is to increase humankind's power over the Universe. Where did I get this goal? Simple; I made it up...where else does anyone come up with goals? Anyway, combining this goal with Point 3 about ontology, I think the aim of science should be to give humankind the ability to accomplish pragmatic things, like predicting future phenomena, or making technology, etc. A secondary goal of science would be for its direct pleasure benefit for non-scientists - e.g., people can read about infinite inflationary cosmology and go "Wow, that is neat-o!" I do not view the pleasure of scientists themselves as a goal of science.

6. Scientific Models. George Box famously said that "All models are wrong, but some are useful." To me, this is like saying "No house is 100% big, but some are big." It's just a silly statement. No model describes all of reality. Most or all models fail to perfectly describe the set of phenomena that they purport to describe. And if some model - say, general-relativistic quantum mechanics - does perfectly describe its chosen set of phenomena,  - say, quantum mechanics, or general relativity - it would hardly be worth mentioning. Combining Points 3, 4, and 5, I think that "useful" and "right" are the same thing when it comes to scientific models. Perfect usefulness ("rightness"?) is only one measure-zero point on a multidimensional continuum of rightness/usefulness.

7. Techniques of Science. It seems to me that all scientific endeavors involve three basic processes: A) Logic, B) Evidence, and C) Judgment.

7a. Logic. Logic, to me, means following some sort of rules for your arguments. I basically think you should always use some sort of logic when you make your arguments, since it seems to help convince people of stuff in a repeated and consistent manner. Other methods of convincing - appeals to emotion, for instance, or tribal affiliation - do also seem to work sometimes, but more sporadically. So I think scientists should always use logic when they can. But logic is like a rule for constructing a chain...it doesn't tell you where to start the chain. You need some sort of premise or starting point.

7b. Evidence. Evidence, to me, means that how well a theory matches past or existing data is an indication of how right/useful the theory is. If science is going to work, then tomorrow is going to be have to be something like today. By Point 4, this means that there have to be some sort of "laws", over some sort of time horizon. Maybe the laws only hold for a short time, but they have to hold for longer than it takes you to figure them out. This means that "induction" is going to have to work to some degree. In other words, how well a model or theory describes data today must be some sort of indication of how well it describes data tomorrow, or else science is useless.

Now here we get to an interesting side question: In what way is a theory's descriptive power an indication of its prescriptive power? You can say "If a theory doesn't match the data, it's not useful/right." This, I think, is what people call "falsification". Alternatively, you can say the converse: "If a theory does match the data, it is useful/right." I don't know a name for that. Actually, I think both these statements are too extreme. Rigid insistence on pure falsification is not always a good idea, since the cutoff for saying a theory "matches data" is pretty arbitrary, like a confidence interval in statistics. And the converse of falsification - "It looks right so it must be right" - seems to lead to overfitting. Also, ranking theories based on how well they fit the data has its own pitfalls, since theories that make big mistakes can sometimes lead to future theories that make fewer mistakes than the best presently existing theories - as an example, Copernicus' initial heliocentric model was less good at predicting eclipses than Ptolemy's geocentric model with epicycles, but it led to the creation of Kepler's theory of elliptical orbits, which predicted both eclipses and planetary phases better than Ptolemy's model.

So in selecting your criteria for matching theories to evidence, you inevitably need to use some judgment.

7c. Judgment. Judgment, to me, basically means using your intuition or instinct to tell you things about the world. Some of this is always present in science, for the reason specified above (deciding how to match theory to evidence). Also, there's at least one other reason to use judgment, since formally there are infinite possible hypotheses to test, and infinite models that fit any set of phenomena. (This is pointed out by Robert Pirsig in Zen and the Art of Motorcycle Maintenance, which is mostly a first-hand account of psychological disorder, but is also a cool philosophy-of-science book.)

So you always need some judgment in science. There are many different ways to use judgment, and the the amount to which you use judgment in each of those ways can vary. A few examples include: What do you set as the null and alternative hypotheses? What kind of confidence intervals do you set for your regressions? Do you toss outliers, and if so, which? How do you penalize the addition of parameters to the model? And, most importantly, there is The Big Judgment Call: How do you use the match between theory and evidence to evaluate the usefulness of a model?

Basically, it seems to me that if you don't have logic, evidence, and judgment, you're not doing science.

8. Different "Scientific" Disciplines. There are many different disciplines that purport to inform us about the world - physics, history, biology, economics, etc. Some of these call themselves "sciences", some call themselves "social sciences" with an emphasis on the "social", and some don't call themselves "sciences". The ways in which these disciplines use evidence and judgment are different. Furthermore, the way that each discipline uses evidence and judgment may change in time (witness the changes in physics between Aristotle and Feynman!). These changes are probably evolutionary, based on trial-and-error - basically, kick your foot into a way of doing science, and see if you stub your toe or not. Aristotle's way of doing physics wasn't useful for producing models that allowed gunners to hit targets accurately from long distances; Newton and Galileo's was. Aristotle's way of doing physics mostly died out, Newton and Galileo's survived and evolved.

Evaluating the usefulness of an approach toward science involves a sort of meta-science, involving evidence about judgment, judgment about judgment, evidence about evidence, and judgment about evidence. Physics seems to have a lot more replicability than history, which is probably why physics has usually relied a lot more on evidence, and history a lot more on judgment. Nowadays, with the controversy over string theory, we see a big debate over whether physics should evolve toward a method in which evidence is less important. In history, with the efforts of people like Daron Acemoglu and Jared Diamond, we see people arguing that history can rely more on evidence than in the past. (Yes, I am using the terms "less evidence" and "more evidence" VERY loosely, see Point 1 about language.)

So each science has its own "scientific method", and this method evolves in time. We just have to figure out what seems to be working and what doesn't seem to be working, and adjust accordingly. Its not always obvious, and it's certainly not a rapid process. We can only hope that the marketplace of ideas promptly produces the best scientific method for each discipline given the available technologies of investigation. But it seems to me, witnessing the failures of science to blossom in the Roman Empire, Abbasid Caliphate, and Sung Dynasty, that opportunities to improve science are often missed.

9. A Few Unhelpful Ideas About Science. I'm not sure if anybody says exactly these things, but they loosely conform to ideas that some people seem to entertain (i.e. they are straw men), so I might as well list them...

"Since all disciplines use judgment in some way, all uses of judgment are equally appropriate." Actually, different uses of judgment sometimes seem to produce radically different results within a discipline, as with the oft-discussed shift in physics, chemistry, and biology in the 1600s and 1700s.

"Science does not always need evidence; sometimes, we can start from judgment and proceed by logic to a conclusion, and then accept that conclusion without checking it against evidence." This is like when the evil wizard tries to win by turning himself into a snake...it never works.

"We only make theories in order to check the internal consistency of our ideas." Who cares? Remember, scientists having fun is not a goal of science (according to my arbitrary value system). We do not pay you $200k/yr to play video games; why should we pay you $200k/yr to satisfy yourself of the internal consistency of your ideas?

"It takes a theory to beat a theory." This may be true sociologically, as a description of how science evolves in practice, but I don't think this ought to be the case. I am fine with doubt and ignorance. I think that bad ideas can block and delay the development of good ideas. And I think that a false sense of certainty can lead to mistakes (e.g. by policymakers).


And, finally, we come to the application of this General Philosophy of Science to macroeconomics. There is not much new in this section; it's a summary of things I've said before, and I just included it here so that you could see how I map from my philosophy of science to my tendency to complain about macro. Since it's such a rehash, this will be my last big complaint session about macro for quite some time.

10. Judgment Calls in Modern Macroeconomics.

I occasionally complain about certain uses of judgment in modern macro. These mostly revolve around one complaint: The macro field does not, in my opinion, use sufficiently stringent criteria for rejecting theories. Multiple theories are simultaneously judged "good" by explaining the same stylized facts (for example, producing simulated economic fluctuations in GDP that match the variance of observed GDP)...These mechanisms can't all be accounting for 100% of the same phenomenon at the same time! Also, macroeconomic models are rarely if ever tossed out because of the results of some statistical test (statistics being the only way we have of matching macro theories to data, since experiments are unavailable). Additionally, the microfoundations used in macro theories are not required to match the microfoundations observed by microeconomists. It thus seems to me that there is "too much" judgment involved in modern macro, and "not enough" evidence. Yes, "too much" and "not enough" are loose terms. 

I suspect that the reason for this, historically/evolutionarily speaking, is the poor quality of macro data. Macroeconomics has better data than history, but not a lot better! You're still dealing with time series that may or may not be ergodic (in other words, macroeconomic history may have just been "one damn thing after another", with no stable "shock process" or "adjustment process"). The time series may not be stationary (unit root tests have low power). Cross-country comparisons are notoriously difficult.

So if you require macroeconomics to hew to the same standards of empirical verification/falsification as, say, tax economics or financial economics, you will be left scratching your head most of the time and saying "Well, we just really don't know what the heck is going on!" So, historically, macroeconomists had to settle for less ambitious goals. They had to behave more like historians, writing "literary" tomes vaguely describing what they thought was going on. After World War 2, this changed, and macroeconomists started to describe their ideas in the language of mathematics. For a while, people thought macro could work a bit like physics, but the Lucas Critque and some notorious policy mistakes seem to have dashed that ambition. Now, macroeconomists seem to be back to "telling stories" (a phrase they themselves often use), though they've retained the language of math.

One common response is that macroeconomics produces a bunch of different models that tell a bunch of different stories, and that judgment should be used to select which stories apply at which times. And I am OK with that in principle! Maybe evidence, rather than judgment, can be used to tell whether, for example, the Diamond-Dybvig model of bank runs is about to come into effect (people like Markus Brunnermeier and Hyun Song Shin are trying to do things like that, which is just one reason why I am big fans of theirs). But the set of possible stories is essentially infinite; it is a certainty that some of these models are bad ones; that they are not as good as they could be, and that evidence could be used to show this and to construct better models. I feel - and this is just the sense I get from talking to people and going to talks and reading papers - that not very much model rejection is being done.

In other words, although I am not a strict "falsification-ist", I think that rejecting models is almost certain to be an essential feature of using evidence to select the best set of models to use in practice.

And what I suspect is that macroeconomics went so long without any hope of matching any data that it developed bad habits. Internal consistency and the collective intuition of macroeconomists were overemphasized, and what little data there was was often ignored. Theoretical tolerance became the norm, and models that were essentially never useful remained prominent in the toolkits of economists and policymakers alike. And the large reliance on judgment seems (unsurprisingly) to have allowed some political bias to seep into the profession.


So to sum up: I don't complain about macro methodology because I have a rigid idea of what "science" ought to be, and I demand that all disciplines either live up to the standards of physics or admit radical ignorance. I simply judge that macro has too much judgment in too many places, that there are popular models out there that could and should be rejected by what little evidence exists, and that many macroeconomists should admit more doubt about our understanding of the "business cycle".

Maybe I'm wrong, and if so, I'm prepared to revise my thinking...

54 comments:

  1. Oooh ooh, first post!!!

    ReplyDelete
  2. Very nice.

    I definitely agree with your pragmatist bent throughout. I think you could probably be a little more partisan on past philosophers given that stance (take your pragmatist attitude to that one... some philosophers are more useful than others!). But I definitely agree with the frustration over arguments from authority.

    ReplyDelete
  3. Anonymous5:44 PM

    "I see absolutely no reason not to reinvent the wheel every time I think about philosophy."

    There are people who have done serious work on (devoted lifetimes to) the Philosophy of Economics. When one expounds on mathematics, or chemistry, etc, does one really not even glance at the literature? Why is the Philosophy of Economics different? It's very unlikely that you will hit on something that has not been said better, or more clearly, elsewhere. I guess that is what philosophers and people in other disciplines find frustrating about economists --- the philosophy etc often seems kinda naive.

    ReplyDelete
    Replies
    1. I agree to a certain extent with this criticism, but the problem with a lot of philosophy of science is that it's often a sort of applied epistemology.

      If you want to understand science itself, you probably need to study it scientifically - and too much philosophy of science is like Noah's "evil wizard that turns into a snake": namely, logic without empirical evidence about how science actually works.

      Some isn't, of course. I'm particularly partial to Kuhn precisely because he conducts himself more as a scientist of science, really investigating the practice of science empirically.

      Delete
    2. Anonymous6:14 PM

      "the problem with a lot of philosophy of science is that it's often a sort of applied epistemology."

      Well, this is an empirical claim...

      One common criticism is to say that economists don't fully realize the extent to which they're doing moral philosophy. On the philosophy of economics in particular: Do you think it is helpful to be familiar with the work of Mark Blaug? Nancy Cartwright on causality?

      Imagine you are thinking about social choice. Would you recommend looking at Arrow before you write a blog post on the topic? Anyway, I think CM below says it better than me. The idea that there is nothing to be learnt from economists/philosophers who have actually worked on the philosophy of economics sounds like hubris.

      Delete
    3. The idea that you come to understand science by studying it scientifically, does not justify ones epistemological assumptions, it nests them within the methodology. If you like Kuhn, you might also like, eg, Popper and Lakatos. If you think some HPS is too far divorced from reality, perhaps you could actually name the authors and works of which you are thinking? This would enable a useful discussion about what should be on our reading list instead of a licensing laziness. That is something I am capable of doing for myself.

      Delete
    4. Feynman was always very good about describing "what science is" and "what isn't science", coming from a very pragmatic viewpoint.

      And Noah's accepted pragmatism (or is it pragmaticism?) as a premise.

      Delete
  4. I write this as a working scientist (well, grad student) in the social sciences, who probably needs to read more HPS himself: I really think that not reading the HPS literature is misguided. It is hard to argue with the exhortation to think for oneself, but it is equally easy to respond with some kind of glib statement like "if I have seen further than other men, it is because I stood on the shoulders of giants", etc etc, ad nauseum. Given that these kinds of arguments don't get very far, consider the following approach:

    1. Think hard about an issue. Come to your own conclusions. Even write down what you think, formalise it, discuss it with colleagues, or put it in a blog post.

    2. Actually go and read HPS, seriously and in depth. Benefit from the insights and ideas of extremely bright people who have thought about the project of science in more depth, with more care and for longer than you or I.

    3. Revisit 1, rinse, repeat.

    Taking (2) seriously does nothing to limit (1) and (3). It only improves ones thinking. Saying "oh, but that disagrees with Kuhn" might be a silly objection, as it seems to rely on an appeal to authority. However, it could also be a shorthand for "the reason your theory is flawed is really complicated and Kuhn addresses it, so I am trying to remind you of that text rather than have a four hour discussion about the issue, because I don't have time or I expect that as an educated person with an interest in HPS and a respect for the work of people in domains other than your own, you would have read it".

    ReplyDelete
    Replies
    1. Saying "oh, but that disagrees with Kuhn" might be a silly objection, as it seems to rely on an appeal to authority. However, it could also be a shorthand for "the reason your theory is flawed is really complicated and Kuhn addresses it, so I am trying to remind you of that text rather than have a four hour discussion about the issue, because I don't have time or I expect that as an educated person with an interest in HPS and a respect for the work of people in domains other than your own, you would have read it".

      Ah, but I didn't say I hadn't read Kuhn, I said I didn't care...

      Delete
    2. Anonymous9:32 PM

      @Noah, just to understand --- your claim is that there's nothing to be gained from reading any economists or philosophers who have worked hard on the philosophy of economics or the philosophy of science? And The Journal of Economic Methodology, Economics and Philosophy, the Erasmus Journal for Philosophy and Economics, the British Journal for the Philosophy of Science, Synthese, and Philosophy of Science are journals of no relevance to any (working) economist?

      I mean, empirically I guess that's how most economists operate (without knowledge of their own intellectual history, and without much economic history either) but has this lack of navel-gazing been helpful? Or is the problem the reverse?

      Delete
    3. just to understand --- your claim is that there's nothing to be gained from reading any economists or philosophers who have worked hard on the philosophy of economics or the philosophy of science?

      Nope! ;)

      Delete
    4. Anonymous9:46 PM

      email maki, hoover, cartwright, rosenberg etc and let them know ;)

      http://www.helsinki.fi/tint/maki/research.htm
      http://public.econ.duke.edu/~kdh9/
      http://personal.lse.ac.uk/cartwrig/Full%20list%20of%20publications.htm
      http://www.duke.edu/~alexrose/ARPublications.pdf

      Delete
    5. Just to understand, your claim is that if some people have done scholarly research on a topic, it's insulting for someone outside the field to offer thoughts on that topic without A) reading and B) explicitly making reference to the work of those scholars?

      Delete
    6. Anonymous10:24 PM

      Not insulting, no. Just, that it would be helpful to read people who have thought about this stuff a lot. You claim that you can think through all this stuff yourself and that that would be quicker than, for example, looking at what Hoover has to say about the philosophy of macro. I think instead that that is unlikely. I'm arguing that this "I see absolutely no reason not to reinvent the wheel every time I think about philosophy" is bad reasoning (if you're interested in knowing the frontier).

      Delete
    7. Just, that it would be helpful to read people who have thought about this stuff a lot.

      Yes, I agree! That will be really cool, and I'm sure I'll enjoy it and learn some interesting stuff. When I get some time, you bet I will read it.

      You claim that you can think through all this stuff yourself and that that would be quicker than, for example, looking at what Hoover has to say about the philosophy of macro.

      You might misunderstand my claim. Thinking through this stuff myself will not reproduce all the stuff those researchers thought of. But thinking through this stuff myself can still be productive! The worst that can happen is that I'll waste time and effort...the best that can happen is that I'll happen to come up with something completely new...

      I'm arguing that this "I see absolutely no reason not to reinvent the wheel every time I think about philosophy" is bad reasoning (if you're interested in knowing the frontier).

      But I'm not trying to push the boundaries of the academic field. I'm just trying to sort out my own existing ideas. That's a far easier task...if I ever write a paper on philosophy of economics, I will of course make sure I have read the literature, just as I would in any field...

      Delete
    8. Noah, I seriously do not understand why you had to write a dissertation. I have told several people that your first days as a Ph.D have been better than the best days of the rest.

      I hope you draw a distinction between citing to authority as a shortcut and citing to authority as proof for in response to some discussion above, for I am going to mention that Richard Feynman explains why experts should listen to lay people in his 7th Cornell Lecture, which is on the Microsoft Research website.

      And, I agree to your response to Anon., who wrote I understand --- your claim is that there's nothing to be gained from reading any economists or philosophers who have worked hard on the philosophy of economics or the philosophy of science?

      Nope! ;)

      Again, only as shortcut for the interested reader, I refer to Feynman who also explains in the same lecture why you are right on this point.

      Delete
    9. As far as I know... nobody can read everything, and I am not sure how many could retain in memory all that they read... so we've a mini-max problem here, am I wrong?

      Nowadays, as far as I know, the custom of having papers or even blog-posts (like on voxeu.org) signed by many people is spreading in almost all sciences, while if I remember well it wasn't like that at the time of Edison-Tesla-Marconi.. so perhaps in this new environment, a fitting new habit and skill for making good research could be that of grouping together people from different sciences, like in those jokes which tell "There's a physicist, an engineer, a philosopher, a historian, an economist, and they're...".
      Chances are that they read a lot of different stuff and possibly made some different experiences at "making science", but when they *discuss* together, they will all just pick from their living memory what it's relevant for them at that moment in order to make others to understand what they mean, so perhaps this could increase the rate of new idea creation and maybe also selection. This for example because when we're trying to explain something to someone who is intelligent and very versed in her field, but has no background in our field, this can help us breaking sparks somehow.

      Delete
  5. "All models are wrong, but some are useful."

    Excellent. That's not silly at all.

    Steve

    ReplyDelete
    Replies
    1. Really? I think it's silly. Dr. Box was being cute when he wrote it, I'm sure.

      The point is, "right" and "useful" mean the same thing when it comes to models.

      Delete
    2. Phil Koop9:22 AM

      I don't think that's true. For example, Newtonian mechanics is less "right" than relativistic mechanics but more "useful" because for almost all practical problems it gets the same answer with less computation.

      Now, if you had observed that you can't talk about something being "useful" until you can say for what purpose, you would have had a point.

      Delete
    3. Phil:

      Of course. If you want to model a cannonball, Newtonian mechanics is awesome. An economist once said "Physicists thought they had it all worked out with Newtonian mechanics, and then it turned out to be wrong, HA!" And then he quoted that silly George Box quote, "All models are wrong". But he was making a big mistake...Newtonian mechanics is right, to a very good approximation, in its domain of validity. More tan can be said for almost any macro theory.

      If you want to make a GPS, though, relativity is more useful. It's "right" within its domain of validity, to a good approximation (maybe an arbitrarily good approximation, butwho cares?).

      So yes, "useful for what" is a key question.

      Delete
    4. Noah - I think Box was saying the same thing as you , but through ironic juxtaposition to make a more memorable chapter heading (btw, the correct quote doesn't have "but" in it). To be pedantic (which is what they pay me to be) right/wrong in common parlance is a Boolean concept whereas usefullness is a continuum measure - which is where your comparison goes by using "100% big" as the Boolean modification of a continuum concept. Seems to me that if 100% big is silly, then 90% rightness is equally silly.

      I think Box was pointing out the absurdity claiming models are "right," because "right" has a common definition that we all understand, e.g. 1+0 = 1 is right and 0+0=1 is wrong. The statement that WWII ended in 1945 is right and the statement that it ended in 1946 is wrong - you can't say that there is some "rightness" to 1946 just because its closer to the correct answer than 1947. Implicitly, these ideas are what the general public understands by right/wrong, so when we say a model is "right," the general public infers different things than our colleagues - which isn't exactly what communication is all about.

      If only George had titled the chapter: "All models are bounded, some are useful," no one would have noticed.

      Of course, Humpty Dumpty would say "When I use a word,...it means just what I choose it to mean..."

      Delete
    5. I think Box was saying the same thing as you , but through ironic juxtaposition to make a more memorable chapter heading

      Of course. Box was being silly, not stupid. He knew how models relate to reality. But lots of people abuse his quote to mean "models should not have to match data for us to use and believe them".

      btw, the correct quote doesn't have "but" in it

      Not according to Wikiquote:
      http://en.wikiquote.org/wiki/George_E._P._Box

      To be pedantic (which is what they pay me to be) right/wrong in common parlance is a Boolean concept whereas usefullness is a continuum measure - which is where your comparison goes by using "100% big" as the Boolean modification of a continuum concept. Seems to me that if 100% big is silly, then 90% rightness is equally silly.

      I really should have said "100% small", since size has a well-defined minimum. Then the analogy would have been apt.

      I think Box was pointing out the absurdity claiming models are "right,"

      Yes, he was pointing out that rightness is a continuum, not discrete.

      BTW, even knowing what it means I still dislike the quote, because even in its proper form, it neglects the situational/conditional aspect of rightness, i.e. the notion of domains of validity.

      If only George had titled the chapter: "All models are bounded, some are useful," no one would have noticed.

      Maybe not. But I'd still be slightly annoyed with the quote, because different models are useful to different degrees for different purposes in different situations... ;)

      Delete
    6. I'm not buying 100% small as fixing your analogy - I can't wrap my brain around the idea that a 100% small house is a house rather than the empty set... try again? ;)

      Delete
    7. No, I think the analogy is apt. You're right that a 100% small house is not a house as we typically think of a house...think harder! ;)

      Delete
  6. "I simply judge that macro has too much judgment in too many places, that there are popular models out there that could and should be rejected by what little evidence exists, and that many macroeconomists should admit more doubt about our understanding of the "business cycle".

    1.Yes, they can all be rejected. They're all wrong.
    2. What are these "popular" models you dislike? Maybe there are reasons these are popular that have nothing to do with your narrow notion of what's useful?
    3. Who's being immodest? I don't know anyone practicing macroeconomics who claims they understand business cycles completely.

    ReplyDelete
    Replies
    1. 1.Yes, they can all be rejected. They're all wrong.

      Reallllllllly? There's no censored data set or subsample in which at least one model holds???

      2. What are these "popular" models you dislike?

      Well, models where supply shocks cause recessions. I would point to them first...

      Maybe there are reasons these are popular that have nothing to do with your narrow notion of what's useful?

      That would pretty much have to be true, I think!

      I don't know anyone practicing macroeconomics who claims they understand business cycles completely.

      Not completely, but I think some of the things people think they understand may in fact be things that they don't actually understand...

      Delete
  7. A good summary, although your attitude to Phil. of Sc. irresistibly reminds one of XKCD #793.

    On the topic of rejection of models: Each theory or model excludes some possible classes of observations, and greater exclusion is a reason for preferring one model over another. (No faster-than-light warp drives or macro-scale wormholes in space are possible in the GR-QM model-plex, for example. That's why space aliens haven't shown up here.)

    Copernicus's theory was preferred by astronomers despite its worse numerical accuracy for this reason: it said certain classes of orbits will never be observed - for example no-one will discover a distant planet that nevertheless appears from Earth to be always near the Sun, as do Mercury and Venus. With epicycles, anything was possible. Copernicans exchanged numerical accuracy and precision for ontological, and made progress in so doing.

    As a criterion for selecting models, ontological exclusion works really well ... until someone makes an observation that contradicts the core ontology of the model, at which point the model has to be rebuilt or discarded. Rejection is painful and mostly prolonged.

    ReplyDelete
    Replies
    1. A good summary, although your attitude to Phil. of Sc. irresistibly reminds one of XKCD #793.

      On the contrary, I think physicists tend to listen to philosophers of science more than they preach at them. Most of what I'm saying is really just recycled Popper, Kuhn, and Feyerabend, as filtered through my undergrad profs, popular authors, and my own brain...

      Delete
    2. Noah,

      I like almost everything you post, which is why I'm surprised you exhausted so much effort to say a lot of things about phil of science written at precisely the tone, demeanor, and depth of the undergraduate commentary you cite.

      I like particularly the idea that we do not prove things in science, rather develop systems of warranted belief. Your warrants above are facile. Though I suppose the point of the essay was to dismiss phil of science as unscientific, and conducting the essay thus in such an unscientific manner conveys the point through practice.

      Delete
  8. Anonymous7:34 PM

    "Ontology is the philosophy of what existence means"

    I guess that's a workable definition but it's more apt to describe one facet of certain types of metaphysical questions within the field of ontology than the whole of ontology and ontological questions.

    A more generalised definition of ontology is about the building blocks or foundations of being, existence and reality and specifically what is the primary locus of constituent parts and inter-relationships.

    Similarly, there are other definitions employed here that are somewhat glib. I think it shows that you haven't read enough in philosophy and philosophy of science.

    ReplyDelete
    Replies
    1. Similarly, there are other definitions employed here that are somewhat glib. I think it shows that you haven't read enough in philosophy and philosophy of science.

      Or that I'm trying to write a blog post not a book...

      Nice try troll.

      Delete
  9. Out of curiosity Noah Smith, what is your opinion of Logical Empiricism, of the type Rudolf Carnap advocated?

    According to Dr. Michael Emmett Brady, John Maynard Keynes was an "early Logical Empiricist" who would provide inspiration to Carnap in the fields of probability theory and philosophy of science.

    http://papers.ssrn.com/sol3/papers.cfm?abstract_id=1920578


    Of course, you don't have to read what Rudolf Carnap wrote or that paper by Dr. Michael Emmett Brady, but you can look up logical empiricism.

    ReplyDelete
    Replies
    1. Zathras10:50 AM

      To this one should add Jaynes, who rigorously systemized these ideas in a Bayesian framework: http://www.amazon.com/Probability-Theory-The-Logic-Science/dp/0521592712/ref=sr_1_1?ie=UTF8&qid=1343314093&sr=8-1&keywords=jaynes+probability.

      Delete
    2. I know that Dr. Michael Emmett Brady considers E.T. Jaynes and Harold Jeffreys to be "logical Bayesians". Here's Dr. Brady's review of that book by Jaynes that you refer to.

      http://www.amazon.com/review/R1EL1ID6WOF4AY/

      Delete
  10. Anonymous10:38 PM

    Falsification is an excellent and efficacious tool. So is a hammer. But, it isn't about "what can I do with a hammer?" It's about "how do we build this house?"

    ps - not to worry - I have cute cat pix!

    ReplyDelete
  11. Mathesmatics
    Science and mathematics are not cool subjects, say students. Consequently, if these subjects are compulsory, students opt for an easier stream in secondary school and are less likely to transition to university science programs.

    ReplyDelete
  12. Actually Noah, the Problem / Paradox of induction is a little more nuanced than you describe.

    True enough, inductive reasoning will allow us to draw the conclusion that the sun will come up tomorrow because it always has. But in fact, our use of inductive reasoning is itself perpetuated by inductive reasoning. We use inductive reasoning because it has always worked in the past.

    We can illustrate the point by showing that the reverse also applies. Lets suppose that we opt for an approach to science based on counter-inductive reasoning. The sun WON'T come up tomorrow because it always has in the past. I can justify holding this belief, because my commitment to counter-inductive reasoning has never worked in the past. I'm still being quite consistent.

    ReplyDelete
    Replies
    1. I guess, sure. But it's like walking into the wall...try it and see how it works, and I'll be over here with a beer! ;)

      Delete
  13. Orange148:24 AM

    Great post and one that I will need to read several times. Wow, someone else who has read Pirsig's Zen and the Art of Motorcycle Maintenance. The book came out while I was working on my PhD in chemistry and I was so taken by it that I used his discussion of 'mu' as the preface to the first chapter of my thesis!!

    I've found that the one general thing to remember in any scientific (social or natural) endeavor is "something works until it doesn't." this is a little more nuanced of an approach than, "All models are wrong, but some are useful."

    ReplyDelete
  14. Anonymous12:46 PM

    I'd say almost all scientists are Popperians and almost no philosophers of science are.

    (Though, when I say "scientists are Popperians" I mean "scientists claim to be Popperians"; their practice doesn't match their theorising for precisely the reasons Popper is not popular amongst philosophers.)

    What would be good would be if scientists and philosophers of science both paid some attention to what the other is doing.

    ReplyDelete
  15. I have been thinking about the same issues after reading your previous posts. And watching Feynman's lectures on youtube.

    I have a clarifying question.
    Where in your system of models are those which are right, but not particularly useful?
    For instance, think of the uncertainty principle (in physics). It is clearly right/true. However, I can't see how it's useful. It's rather restricting us in all kinds of dimensions.

    There are a lot of results in economics, which have the same flavor. For instance, stock returns may be inherently unpredictable because, if they were somewhat predictable, somebody would have made money by betting on that information, thus, making them less predictable. I feel that the number and effect of these kinds of right/useless results on stuff studied by economics is much bigger than on stuff studied by physics.

    ReplyDelete
  16. Anonymous9:18 PM

    There are several comments about how Box was being silly or absurd. I do not think he was. He meant that models are not reality. Noah says (I think it was Noah, in response to another comment) that Box knew the relation of models to reality. What does Noah mean by that?

    I believe that Box meant that there is no relation between a model and reality. We often confuse our models with reality, but they are not real, only useful. When you learned the structure of the atom in freshman chemistry, did you think that an atom was like a little solar system? That is not at all what the structure of the atom is, it was only a useful fiction that fit a lot of facts. There is a newer model of atomic structure with weird shaped d and p orbitals, dumbbells and tori... do you think that is the actual structure of the atom?

    Of course, as science progresses, it seems that our models get closer to "reality". My belief is that it only seems that way. It's our hubris. Right now, in our time, we believe that light is both a particle and a wave. We believe that matter has wave properties. Think about that. I mean, really? We have no idea what light is, what energy is, what time is. Basic things about the world we live in. One model of electrons, that they are particles, fits some data, while another model, that electrons are waves, fits other data.

    Do you know what the ultraviolet catastrophe was? Back in the day, the Rayleigh-Jeans law fit some data about "black body" energy emissions, while the Wein approximation fit other data. There was no way to reconcile the two models. Then Planck's law came, and explained the spectrum of such radiation very well. Of course that implied that energy was quantizied (?sp) and led to quantum theory.

    Do you believe that quantum theory is more real than the classical models it replaced? There are still some things that quantum theory does not explain in chemistry. It is very useful, but is it right?

    When a theory comes of matter and energy, that can reconcile particle/wave, will it be right?

    Models are "wrong" in the sense that they are not actual reflections of reality. I understand Noah's pragmatic sense of the world. But, don't let it limit you. You may stub your toe on a wall. Trying to disbelieve that the wall isn't there will not keep you from stubbing your toe. On the other hand, Dont let your idea of a wall be set in stone (as it were). If you lack imagination you might not see the door in the wall, that if you open it, lets you magically go right through.

    This is my view of what George Box meant. This is not silly.

    ReplyDelete
  17. TL;DR:

    RBC models are bad, man.

    ReplyDelete
  18. Re the Kuhn/Popper tagline.....is it because they found the column shifty and unfalsifiable?

    ReplyDelete
  19. I tried to stay away but I just couldn't miss the party!

    An acquaintance of mine, a Philosophy Ph.D. from Oxford, always said that first you define your terms, then you start debating them. It saves you a lot of trouble, like finding out after several hours of debate that you were talking about completely different things.

    Kuhn is interesting because, as I believe CM pointed out, he wrote from a descriptive and not prescriptive perspective. This is how science really works. Of course this does not mean it should work like that, but given its contributions to our standards of living any challengers will have to put on a great performance to convince us otherwise. At this point I want to mention here the book "Better living through economics" that I cited in SW blog. It contains examples where economics have really made a difference.

    I am hesitant to assign a "goal" to science. History teaches us that significant scientific discoveries can come from research agendas that seemed obscure at the time. Therefore, I side with science for science's sake (hey, if it's good enough for art...). Otherwise we are openning the door to a very real threat of someone somewhere deciding what types of research should and should not be undertaken (I remember someone on Fox news being very upset that tax money funded a project they considered "ridiculous", researching whether women respond sexually to images of porn).

    Going back to Kuhn (and Sokal), their point is really on target. A general theory makes several predictions. Suppose that you collect data, you construct your arbitrary confidence intervals, you throw out what your judgment says are outliers, and you test the set of predictions. For some the results are qualitatively and quantitatively consistent, for others the sign is right but not the magnitude, and finally for some the sign goes against the predictions. What do you do? Do you throw the theory out? Kuhn's point is that if that was common practice we would have no theories ever, every one of them would have been thrown out. What people do instead is look at the successes. If the successes are enough to warrant it (and this judgement is ALWAYS subjective, no confidence intervals here) then they continue to work with it. They examine if there is a mistake in the testing, or try to fine-tune it by reforming some of its components (e.g., introducing dark matter, or adding frictions to the labor market in RBC models like in Blanchard and Gali (2008)) and see if the predictive power improves. This is what Kuhn calls normal science. Continuous failures to do so and a growing dissatisfaction will of course increase the number of scientists pursuing alternative venues. If they succeed in formulating a more predictive "paradigm" then the old paradigm is dropped in favor of the new one, but only then. Well, I am perfectly happy with that!

    ReplyDelete
  20. Correction: Blanchard and Gali is in 2010

    ReplyDelete
    Replies
    1. An acquaintance of mine, a Philosophy Ph.D. from Oxford, always said that first you define your terms, then you start debating them. It saves you a lot of trouble, like finding out after several hours of debate that you were talking about completely different things.

      But it's hard to do, because language is imprecise, and terms can only be defined in terms of other terms. For example: What exactly is the definition of "flying"?

      I am hesitant to assign a "goal" to science.

      I don't...I'd be more hesitant to exclude goals from science, but I do that too...

      Re: Kuhn...sounds a lot like what I was saying...coincidence?? ;)

      Delete
    2. Precisely because language is imprecise (pun not intended), it is important that those who debate flying spend some time to define its meaning for the purpose of the discussion. This need not be a universal definition. It is for the purpose of the discussion only.

      Delete
    3. Science is imprecise. Narrowing to tolerances is worth the candle, and we can accomplish that in defining terms.

      Delete
  21. Wow, Noah, here and in your 1) you've hit on an incredibly important point that actually Karl Popper made back in the forties against practically the whole philosophic tradition: that arguments over terms are logically irresolvable. Yet somehow this pointless exercise has come to dominate philosophy (Popper sheets responsibility back to Aristotle's methodology). He also contrasts it with how little terminological disputes there are in science:terms are simply for convenience, and argues that much of the progress that has been made in science has been by abandoning the ancient Aristotelian method. If you're interested I can give you a link to his classic, and much overlooked, essay on the subject.

    ReplyDelete
  22. This looks like an excellent place to hop on, so I'm hopping. Thanks for a great, clear, definitional post.

    ReplyDelete