Monday, July 01, 2013

Four levels of science

More philosophy-of-science blogging!

If you haven't yet read "Tantalus on the Road to Asymptotia", Ed Leamer's recent essay, and if you're at all interested in statistics, empirical economics, or science in general, then you should go read it. The essay is primarily a reply to an extremely important 2010 discussion paper by Joshua Angrist and Jorn-Steffen Pischke, called "The Credibility Revolution in Empirical Economics". That paper in turn is mainly a response to a 1983 Leamer essay called "Let's Take the Con Out of Econometrics". Such are the time scales over which deep academic debates are conducted. Actually, you should read all three.

The more recent two essays are discussing the idea of "natural experiments", and to what degree these make empirical economic studies (econometrics) more reliable. This is a very deep question about science. Normally, statistics can only see correlation, not causation; for example, you see that every time roosters crow, the sun comes up shortly afterward, but this doesn't tell you which caused which. A "natural experiment" would be if, for example, some disease killed all the roosters in town. Seeing that even without roosters, the sun still came up, you could conclude that rooster crowing (or, at least, rooster crowing in this specific town) was not necessary to summon the sun.

This natural experiment is very similar to a lab experiment. In fact, how is it different? Well, you might say, a lab experiment is controlled, and a natural experiment is not; in a lab, you can make sure outside stuff isn't disturbing your setup, while in a natural experiment you can't. This, in fact, is Ed Leamer's critique of natural experiments.

But I'm not sure that's right. In a lab experiment, we only convince ourselves that we've excluded all the outside causes. But sometimes, stuff that we didn't think about is messing with our experiment - cosmic rays, or the composition of the air, etc. Sure, lab experiments tend to exclude a lot more causes than natural experiments, but this need not be the case. For example, in finance experiments, even if you use an incredibly simple asset-market setup, subjects' behavior may be distorted by their pre-existing beliefs about the real-world stock market.

As I see it, the biggest advantage of lab experiments is that you can do them many times. You just can't do that with natural experiments. First of all, that allows you to control for a lot more things, since any confounding influence would have to be constant over space and time. Second, you can generate as much data as you want, making small-sample problems (another problem noted by Leamer) irrelevant. And third, it allows you to vary the setup intentionally, exploring the scope of an effect or a theory, and gaining a more complete picture of how the thing works. In other words, in even a perfectly designed natural experiment, we don't get to choose the questions we ask the world. In the lab, we do.

Ed Leamer focuses on the effect of confounding influences in natural experiments. He suggests doing sensitivity analysis to find what assumptions and specifications you need for a result to hold. I think that's a good idea. Basically, it's a way of groping around for something that looks like a set of scope conditions - testing the hypothesis to failure, to get a slightly better idea of when your theory works and when it doesn't. (Of course, if the natural experiment were a lab experiment, you could do this infinitely better, but if wishes were horses...)

As I see it, there are basically four "levels" of science. Each level gives you more confidence in your understanding of the world (i.e., in your theories and models). The levels are:

Level 1: History
This is basically just establishing precedents. It helps you define the set of things that can happen. Imagine a world without writing, and you'll see how important history is.

Level 2: Non-causal statistics
This is basically hunting for correlations. It can help you generate some guesses and ideas about what might cause what. It can also throw cold water on existing theories, since if A causes B, then we should probably see some kind of correlation, however variable or out-of-order, between A and B.

Level 3: Natural experiments
This is when you have some sort of randomized variation, but no ability to control the environment. An ideal natural experiment lets you establish that a causal effect occurred, but it's very hard to tell whether the setting was ideal or confounded, and you get only a limited amount of data.

Level 4: Lab experiments
By allowing replication and control of the environment, lab experiments usually produce more convincing  conclusions about causal effects, generate as much data as you want, and allow you to explore the scope of the scope of the effects you find (i.e. when they do and don't happen).

If we could always understand the world through lab experiments, we would. When we can't put things in a lab - like the macroeconomy, or the Milky Way galaxy - then we should look for natural experiments. But if we can't find sources of random variation, then we should at least look for correlations. And if we don't have reliable quantitative data, the best we can do is just write down what we see.

Update: Noah receives a partial smackdown from...Noah's dad! The father is not satisfied with my one-dimensional classification of research methods, and wants to bring external validity into the picture:
Briefly, research methods vary on two important dimensions, one we can call internal validity (how sure are we that we know what caused our results?), and the other ecological validity (do our observations relate to the real world?). Only the experimental method can logically show cause-and-effect, so it is highest in internal validity, but the artificial situations created by controlling so many factors make it low in ecological validity (also, experiments can be flawed in many ways, such as poor methods, restriction in the range of observations, confounding factors we didn't think about, etc., which is why replication and attempts to falsify claims are intrinsically important to experimental science). Naturalistic observation is highest in ecological validity, lowest in internal validity. Other methods, such as correlation, ex post facto "experiments" (aka, "natural" experiments), and case studies are in-between on both dimensions.  
Even experiments can vary on these two dimensions, some tightly controlled and measured, some using more naturalistic real-world manipulations and more complex settings in which many factors can interact. The ideal situation is one in which experiments at both ends of this continuum show the same thing, thereby bolstering internal and ecological validity. I refer to this approach as "alignment" in a research area, which helps tie real-world phenomena to causes. This ties, for example, my highly controlled lab research on creative cognitive processes (like fixation or incubation) to more naturalistic research with design students, and to research with real designers with real jobs.
Well, there you have it. Note that I was originally trained as a physicist, and in physics, the Principle of Superposition assures you that any conclusion with internal validity will have external validity as well (i.e., the real-world motion of objects is just assumed to be caused by a straightforward combination of things that you can observe in labs). This is less so in other sciences, and much less so in social sciences like psychology and econ.


  1. Interesting. I read Leamer's "Tantulus" more as a critique of being satisfied with consistency even in the face of a small sample (As Angrist advocates in "mostly harmless econometrics", instead of working to get a more efficient estimator right.

    1. He does talk about small-sample bias, but a lot of his essay is about the presence of confounders (which interacts with the small-sample problem). Lab experiments, which can be replicated, basically don't have small sample problems. I'll add a sentence about this.

  2. Anonymous8:33 PM

    3 and 4 map to experimental design concepts of correlative designs versus manipulative designs. A disadvantage of correlative designs is a difficulty in establishing cause and effect and rejecting reverse causation.
    Manipulative designs are better for establishing cause and effect because the variables can be controlled independently.

    Much of economics is built on correlative studies. Game theory is one way to introduce manipulative experiments. A science can progress with both types of information, but the experiments need proper design and the limitations on the conclusions understood.

    -jonny bakho

  3. Anonymous9:00 PM

    What about field experiments? Better than natural experiments and maybe even better than lab experiments?

    1. That's a great question. I'm not really sure!!

    2. Anonymous2:05 AM

      Depending on who you ask, field experiments are either the best of both worlds (control over design and policy intervention with realistic environments) or the worst of both worlds (small sample size due to cost, with the interventions adding an element of 'fakeness' and demand effects that could bias behaviour).

    3. Field experiments have one advantage over lab experiments, i.e. the potential for greater external validity (because the setting is more "natural" than a lab). And unlike natural experiments, you can ask the question you want.

      However, field experiments are less controlled than lab experiments; that "natural setting" might turn out to have been unnatural in ways that it is extremely hard to predict or assess. Also, as you mentioned, the sample size is typically small. And finally, replicability is expensive and difficult and much less reliable than lab experiment replication.

    4. Nathanael1:47 AM

      I personally think that the best experiments are those conducted by people like Derren Brown, who get individuals to volunteer, or even pay, to have psychological experiments performed on them, including ones which would be deemed unethical if done in an academic context.

      Consider how to apply this to economics please. :-)

  4. Ludwig von Mises11:29 PM

    Level 5: Reasoning from undeniable a priori truth.

    1. You're being flippant, if I may say, Noah.

      I would state it differently. We don't have enough undeniable a priori truth (first principles) to make much progress.

      Even something as simple as "when item X gets cheaper, people will buy more of it" doesn't always hold. Or "when something gets more expensive, people will seek to produce more of it".

      If the laws of supply and demand are not constant, what in the world of economics is an undeniable a priori truth? I can't quite think of one... Except, maybe, "people don't like to be robbed and will eventually take steps to prevent it from happening". Not as silly as you'd think since this explains why people trade rather than pillage each others. It also explains why it is better to not impugn too far people fuzzy sense of fairness.

      What other undeniable truths are there?

    2. I deny that there are undeniable a priori truths.

    3. I will say, from the comment (and the blog name chosen) it is impossible to see if this is meant to be serious, or meant to be satire. Of course reasoning from a priori "truths" is not science.

    4. "Of course reasoning from a priori "truths" is not science."

      Nonsense. Michael Polanyi demonstrates the contrary very well in _Personal Knowledge_.

    5. Ok - I'll rephrase it, it is not what I call "science".

    6. reason wins

      ...unless Ludwig is a joke, in which case Ludwig wins

    7. "Ok - I'll rephrase it, it is not what I call "science"."

      Who cares what YOU call science? Polanyi discusses this fact in reference to crystallography and relativity: everyone else calls them science, so I don't think we should worry about whether or not you do.

    8. Anonymous1:06 PM


      Do what now? Crystallography and Relativity (another example would be Maxwell's equations) aren't impressive because they're a priori reasoning, they're impressive because THEY WORK. And we know they work not because of a priori reasoning that they should work, we know they work because they hold up to strict experimental testing intended to show that they don't.

      It's the testing and empirical verification that makes them science. Otherwise Einstein and Maxwell would be no different than any crank on the internet with an "alternative" theory (or Ron Paul libertarian...).

    9. I love making troll threads on Noahpinion posts...

    10. ARGH!

      I got trolled... ;-)

    11. Nathanael1:45 AM

      Reasoning from undeniable a priori truth is great and all, but the only results you get are results in pure mathematics -- tautologies. You can't say anything about the real world.

      This is because the only undeniable a priori truths are *logical* truths.

      Everything else -- even the principle of continuity, which underlies most of empirical science -- are only working assumptions, subject to being disproved. (By the "principle of continuity" I mean the idea that things are basically governed by the same physical laws today as they were yesterday.)

  5. To me the biggest obstacle to economics becoming a science is the measurement problem. Morgenstern's On the Accuracy of Economic Observations talks about this. In physics and engineering there is always measurement error, but the errors are normally distributed and thus the mean can be estimated. There is no evidence that the same is true in the field of economics. Something as basic as the price of an article cannot be measured with precision and may not even exist.

    It follows there are no true functions in economics in the mathematical sense, let alone continuous ones. The use of differential and integral calculus is therefore spurious. Even simple algebraic equations with = signs should not to be taken seriously except as a heuristic devices to give one an intuitive understanding of certain general relationships such as the "laws" of supply and demand or the quantity theory of money.

    Does this mean there is nothing left to study? Not at all. In the first place the law of diminishing returns in its many guises is real and no doubt has a physical basis that should be amenable to scientific investigation. It probably boils down to a problem of physics and chemistry which explain why it is impossible to grow enough food to feed the world in a flower pot no matter how much fertilizer we add. Something similar probably underlies the diminishing marginal utility of income, the increasing marginal disutiliity of labor, and so on. But when these relationships are drawn on a blackboard with a piece of chalk it is not the tip but the side of the chalk which should be placed on the board. The convexity and concavity of the relationships are what we need to understand, for which "less than" or "greater than" but not "equal" are the appropriate signs. Macro-economic statistics are also valuable aids to understanding for all their imprecision, as are game theory and experimental micro-economics.

    So if economics is not a science, what is it? Economics is a logic and an art -- an art informed by logic -- which in the hands of an experienced statesman with a knowledge of economic history and the history of economics can provide guidance in the design of good policy. Ben Bernanke is a good example of such a statesmen in my opiniion, as was Paul Volker before him. Reagan came along at the right time, too, when his Democratic opponents doubted something as fundamental as that a deregulation of natural gas prices was likely to lead to an increase in supply.

    Economics explains why in an open market economy with good information there is a tendency towards (an ever shifting) general equilibrium which represents an efficient utilization of resources, and why this efficiency is a necessary though hardly a sufficient condition to maximize the general welfare. It shows us why Adam Smith was basically right in other words and why Ricardo's theory of comparative advantage is real -- but only if you take its "principle of compensation" seriously when the advantage lies in vast disparities in the relative endowment of the factors of production (as in the case of our with China today, for example, which is undermining the living standards of our once middle-class).

    I'm an old man now, These are the lessons I've learned or at think I've learned from half a century of study and close observation. But I've rattled on too long already. Let me close by saying that in the final analysis economics is for statesmen, legislators and the men who advise them. It should also be an important part of the general education of intelligent citizens in a democracy who (hopefully) will elect better ones when they grow to maturity. What the world needs now is more and better teachers of economics with good Master's degrees, not more PhD's playing with mathematical models.

    1. If marginal dis-utility of labour was always true, how do you explain that some people take pride in working longer hours than most anyone else?

    2. "an efficient utilization of resources, "

      How do you define "efficient" here? I think it is very hard, and that is a major issue. (Think about motor sport for instance. Or even golf.)

    3. In fact I think many of the major issues in economics are hiding in plain sight, by distorting the meaning of commonly used words (enabling all sorts of misleading rhetoric).

    4. "So if economics is not a science, what is it?"

      I think the word you are looking for is "humanity." If economics is not one of the social sciences, then it is one of the humanities.

    5. Bill Ellis8:35 PM

      I disagree with your first sentence. To me the biggest obstacle to economics becoming a science* is that it is humans studying humans.

      Humans are fundamentally game players and the winners will change the rules as they like. If we did have really good measurements that allowed us to reliably predict our behavior that information would be used by some to take personal advantage of the situation causing others to change that behavior.

      We are Heisenbeings .

      * It depends on what you mean by science, Econ is a soft science .

    6. They used to call it a moral science. I always liked that description. But then they also called economics political economy. The emphasis was on policy, on the material interests of people. Browsing through my old copy of The New Palgrave Dictionary of Economics I was amused to read: "Neoclassical economics sees the delivery of individual consumption as the main object of the economic system."

    7. "Morgenstern's On the Accuracy of Economic Observations talks about this. In physics and engineering there is always measurement error, but the errors are normally distributed and thus the mean can be estimated. "


  6. In the physical and biological sciences, "generate as much data as you want" does not always hold. Its sometimes "generate as much data as you can afford," when the costs of running the experiment are high due to material costs or limited access to equipment (think about time on a superconducting supercollider). Sometimes its "generate as much data as you have patience for," when the duration of the experiment is long. Doing NMR before the advent of superconducting magnets, it sometimes took me a week to get a single data point.

  7. Noah: "Normally, statistics can only see correlation, not causation; for example, you see that every time roosters crow, the sun comes up shortly afterward, but this doesn't tell you which caused which"

    I am not sure where you got this, but it is actually relatively simple to discern the direction of causation of two phenomena - by using another independend variable. See more here:

    1. You may be surprised to learn that kind of analysis in economics has a name. The name is "natural experiment".

    2. Yes I understand. My point was that even natural experiment can provide enough information for us to show the direction of causation of two correlated phenomena observed in nature. You don't "need" to change the circumstances in lab to have this.

      That is why I quoted part of Noah's blog as he is wrong about what statistics can and cannot do if filled with data from natural experiments.

    3. Except that Noah specifically uses the word "Normally" to distinguish statistics in the absence of natural experiments from statistics with natural experiments. "Normally" means level 2; you are talking about level 3.

  8. "Normally, statistics can only see correlation, not causation..."
    This is "normal" only in economics. In physics, the first rule is - there is no causation without correlation. Physics students must learn statistics before they do labs. Otherwise, no result can be understood and interpreted. At the end of the day, true physics is a number of statistical relationships with the uncertainty estimated from the experiment (lab or natural) setup. This uncertainty is the future for physics.

  9. Anonymous5:42 AM

    Nice to see you tackle some philosophy of science stuff Noah.
    Although I agree with much of what you say, I think you have a to rosy view on the the lab. May I self-indulge and be allowed to recommend reading some of my own stuff, I would suggest that you read
    (I have a longer article on the topic in the next issue of Real-World Economis Review, that is published later this week).

    1. Yeah, I didn't talk about external validity, but maybe I should have. See the update by my dad, in which he sets me straight...

    2. Anonymous5:37 AM

      As a father of five, I can't but agree - dad is always right :)

  10. Your post seems to presume that assessing causality is the goal of all science. That may be, but causality can be pretty fuzzy concept itself, as I discuss here:

  11. Level one is, in fact, the deepest, highest confidence level of understanding of social life. The other "levels" are abstractions from history, and thus necessarily falsifications of it.

    1. Your understanding of the word "deep", and mine are clearly completely different.

  12. There's one I would put in between your two and three, and that's modeling from first principles. It's too important to be left out. When you have a working model you can learn something whether your hypotheses are proved or disproved by experiment. In my opinion model development and observation are the most difficult parts. Cheers!

  13. Funny, I just watched Pischke give a talk at LSE. He debated another professor, Vassilis Hajivassiliou.

    To be honest, I think Hajivassiliou. His basic point is that we shouldn't run around looking for natural experiments to inductively determine how the world works. We need to first use deduction to come up with some hypothesis about how the world works, and then natural experiments can serve as *one* useful tool to test our hypothesis. But there are many other tools, including lab experiments, where possible, and longitudinal techniques.

    Hajivassiliou's worry is that the experimental craze that's taking over econometrics may end up crowding out the teaching of theory in the classroom. He should be rigorously studying the assumptions behind OLS, for example, so that we will know if OLS is an appropriate approach for whatever we're trying to estimate. More often then not, however, there are non-linearities in many real world phenomena, making OLS a poor estimation tool. In that sense, we all need a better theoretical understanding of how to deal with non-lineariites - and, no: simply adding quadratics and cubics into OLS regressions doesn't appropriately solve the problem.

  14. From the paper:"Though many members of our profession have jumped up to support
    the $787 billion stimulus program in 2009 as if they knew that was an appropriate
    response to the Panic of 2008, the intellectual basis for that opinion is very thin,"

    If that's true perhaps Economics just doesn't know anything (or will never know enough to justify policy) or maybe, just maybe, the discounting of positions as "thin", as econ SOP, is applied for irrelevant reasons (like political or methodological bias).

  15. Hey Noah,

    Have you read Philip Mirowski's 'More Heat than Light'? I'd be very interested to hear your thoughts on his thesis: that 'neoclassical economics' is strongly influenced by an attendent desire to imitate physics.

    Personally, I'm fascinated by the number of economists who write on the philosophy of science -- Jevons's 'Principles of Science' is an especially remarkable Victorian artifact. He apparently spent more time on that work than his 'Theory', and considered it to be of greater importance.

    Anyway, I enjoy your blog (+Twitter) tremendously. Always thought provoking.


  16. I wrote something similar some time back in What evidence is convincing. I didn't have references to this Leamer dialogue though.

  17. Bill Ellis9:07 PM


    You often take shots at Econ for pretending to be more of a science than it has a right to. ( It is not so good at predicting stuff is it ? )

    We know it will never be physics but I always wonder, how do you rank Econ among the "soft sciences " ?

    Here is my ranking of Sciences, hard to soft...

    1) chemistry/ physics

    2) biology

    3) psychology

    4) sociology/economics

    5) social sciences / political science.

    I put Econ tied at the 4 spot .

    They are all useful. But the softer the science, the more it seems to be prone to generating irrational hopes... in experts and laypeople alike.

    1. Bill Ellis - is ecology part of of biology?
      Because I think economics should be a sub-discipline of ecology. That it isn't says a lot about human hubris.

    2. P.S. I think you cannot really rank sciences from hard to soft unless you subdivide them. Is cosmology really a hard science?

  18. This is an excellent post, Noah Smith. However, I think it might have benefited if you had read this book by the late David Freedman, who was a Professor of Statistics at UC Berkeley.

  19. With Physics & Chemistry (and by extension Engineering) there are fundamentals which are undisputed (at least on the level they are being looked at). There is usually a theoretical story to go along with an experiment, and some rationalization of the results. Plus, you have physics, chemistry & engineering MAKING USEFUL THINGS.

    My family likes to say that physicists aren't 100% sure how flying happens. I say, maybe not 100%, but close enough that they make safe planes!

    I think you have to look at the results from economists' recommendations. Austerity has not helped, by any measure. Deregulation produced significant problems. Expansive monetary policy has not been inflationary (imagine if mortgage rates had been just 5% over the past 2 years - where would housing be then?). Then there are the pure political projections - sell stocks in 2009, they're going further down on fears of the Obama. You can look at how predictions fared, and then what went into the prediction.

  20. Regarding lab experiments, Noah may want to consult Latour and Woolgar's classic "Laboratory Life" in which the actual messiness of science in biology labs is described. It's just a small step from the old point about the high school chemistry labs in which no student ever gets the actual predicted outcome of any experiment, but chemistry teachers normalize results through a variety of techniques that preserve the validity of chemistry's theoretical foundations!

  21. Anonymous9:10 AM

    "Science" is based on assumptions, called "laws", which do not fail when experiments are run (neither natural nor lab experiments). These laws allow us humans to manipulate in order to reliably get desired results (put on the gas, the car goes faster; almost always).

    The goal of economics "as a science" is to create policies with predictable outcomes, reliably.
    Policy can be political, or individual, or any organization (i.e. fund manager, any business).

    The failure is in the reliability. The desired economic outcome is, in most cases, a higher than average return on investment (of capital or labor or both).

    At a company level, a huge policy is wages -- how much to pay to the employees for "the best results" (usually net long term profit).
    Another policy involves pricing; another is purchasing; yet another is product development; and of course marketing.

    Most blogs focus on political policy -- what are the best gov't policies?

    The problem in all cases is that the decision makers choosing the policy are manipulating human agents in order to maximize some benefit for the policy decision maker, but that benefit is seldom the same goal as the human agents have for themselves.
    Unlike gasoline molecules who always accept being blown apart for the purpose of powering a car, humans will choose differently over time.

    It might be that a policy maker finds an economic "law" which works.
    When that policy is chosen, it manipulates the agents in some way other than the optimizing of the agents, which the agents learn about over time. And the agents change their behavior. And the "law" fails, because it was only really an assumption.

    [Such as ... US house prices always go up. ]
    Any discovered economic law (=assumption), that can be used by some in order to make more money, will be used, which will change behavior such that the assumption fails.

    Thus economic science can never be successful over the long term at the type of manipulation that non-human sciences can aspire to, so it is better to not even call economics a "science" -- and lots of math is used to obfuscate this truth. Which more humans are understanding, tho there are still many who don't.

  22. Noah: "Well, there you have it. Note that I was originally trained as a physicist, and in physics, the Principle of Superposition assures you that any conclusion with internal validity will have external validity as well (i.e., the real-world motion of objects is just assumed to be caused by a straightforward combination of things that you can observe in labs). This is less so in other sciences, and much less so in social sciences like psychology and econ. "

    Another way to put it is that it took physics a loooooooooong time to get to the point where they could make statements like that, with a decent chance of being right (remember, 'objects in motion tend to come to rest' was a physical law for a long time).

  23. According to your scheme, a randomized trial, eg in medicine, is a "natural experiment" because it is closer to "some sort of randomized variation, but no ability to control the environment" than it is to "control of the environment" in a lab setting. Deliberate and systematic randomization may turn out to be the greatest advance of the 20th century, but it remains underused, even in its stronghold of medicine, and is neglected altogether in your scheme. The core is that you only need to control ONE THING, but if you randomize systematically, and have enough data points, everything else, while not physically controlled, will cancel out.

  24. Excellent intellectual article. But I have some comments.

    Although history is a kind of first dimension of some perfect typologies, I would suggest that in the case of economics and science there are actually other better-preferred choices.

    For example, where there is logical validity, the dependence is on logical coherence, which tends to be divided into formal and applicational coherence. This distinction is especially important in the upcoming era in which people more and more often rely on computing systems for their daily significance.

    Other forms of coherence exist which are also analogical with history, e.g. coherent experiments (physics) and coherent perspective (biology).

    Whatever the taxa, it is important to realize what the highest-tier element is, before reducing the other elements to variations of an unknown. Of course, typically it IS known to some degree within the discipline, in some ways there is no avoiding this.

    But the interdependence of the systems depends on the common factors which have been established as first-tier methods. If experimentation is against perspective, or if logic is against experiment, then there must be some means to combine the disciplines, or some of the coherence is lost. OR it could be argued that decoherence is a virtue, as has been cited in economics.

    So, there is an initial pattern of coherence / decoherence, and a secondary pattern of formal / applicational. It becomes important to develop one of two things: [1] Formal Coherence, or [2] Incoherent Applications, otherwise Formal Incoherence results. But there is one exception (the remaining category): Coherent Applications. This is an understated claim among all the sciences.

    My book on coherent systems is highly recommended. It is called The Dimensional Philosopher's Toolkit (not the original Philosopher's Toolkit). It was published earlier this year.

  25. Thanks you. This was a great pair with this article on the difference between science and engineering.

  26. A lot of sciences rely on natural experiments. Look at evolutionary biology or cosmological astronomy. Careful observation tends to reveal patterns. Look at chemistry. All those experiments studying atomic weights and elemental properties revealed the period table of the elements.

    The problem with economics is that it works in supreme isolation. It completely ignores accounting, which is like a chemist ignoring energy conservation. It completely ignores anthropology, which is like a biologist ignoring zoology or physiology. It's just that economists ignores the real world, but they ignore what we already know.

  27. Nathanael1:40 AM

    Worth noting, Noah, the Principle of Superposition does not apply to large portions of physics. (Because large portions of physics are nonlinear.)

    Ask any astronomer: it matters where your observation point is. A lot!