Tuesday, October 14, 2014

Three ways of understanding the world

P-E Gobry has a narrow view of what science is:
[L]et me explain what science actually is. Science is the process through which we derive reliable predictive rules through controlled experimentation. That's the science that gives us airplanes and flu vaccines and the Internet... 
Countless academic disciplines have been wrecked by professors' urges to look "more scientific" by, like a cargo cult, adopting the externals of Baconian science (math, impenetrable jargon, peer-reviewed journals) without the substance and hoping it will produce better knowledge.
Predictably, one thing this leads to is the conclusion that economics isn't a science:
Since most people think math and lab coats equal science, people call economics a science, even though almost nothing in economics is actually derived from controlled experiments. Then people get angry at economists when they don't predict impending financial crises, as if having tenure at a university endowed you with magical powers.
If you want a rebuttal to the "econ isn't a science" thing, Adam Ozimek has one here.

But I want to make a different point. I think Gobry is right that there's something special about controlled experiments, whether or not you want to restrict the word "science" to mean only that. But there are other ways of systematically understanding the world. In fact, I think there are 3 big ones:

Method 1: History

One way of systematically understanding the world is just to watch it and write down what happens. "Today I saw this bird eat this fish." "This year the harvest was destroyed by frost." "The Mongols conquered the Sung Dynasty." And so on. All you really need for this is the ability to write things down.

This may sound like a weak, inadequate way of understanding the world, but actually it's incredibly important and powerful, since it allows you to establish precedents. What happened once can happen again. Maybe you don't know why, or how likely it is, but you know a bad harvest or a Mongol invasion isn't out of the realm of possibility, and that is valuable knowledge.

Method 2: Empirics

A second way of systematically understanding the world is repeated observation. This is where you try to make a large number of observations that are in some way similar or the same, and then use statistics to identify relationships between them. This is most (though not all) of how economists understand the world.

The first big limitation of empirics is omitted variable bias. You can never be sure you haven't left out something important. The second is the fact that you're always measuring correlation, but without a natural experiment, you can't isolate causation.

Still, correlation is an incredibly powerful and important thing to know.

Method 3: Experiments

Experiments are just like empirics, except you try to control the observational environment in order to eliminate omitted variables and isolate causality. You don't always succeed, of course. And even when you do succeed, you may lose external validity - in other words, your experiment might find a causal mechanism that always works in the lab, but is just not that important in the real world. This is a big big problem for psychology, including prospect theory. 

Experiments give you information about mechanisms. When these mechanisms have external validity - for example, when the same process that moved balls down ramps on Galileo's desk happened to be the one that moves the planets in their orbits - then experimental science (what Gobry just calls "science") is incredibly powerful, more powerful than either of the other techniques. But it doesn't always work.

You may be thinking: Where does theory fit in with all this? My answer is that theory is part of all three of these. Theory is needed to understand causal mechanisms found in experiments, to explain correlations found with empirics, or to isolate the important features of a historical event. Sometimes the theory comes before the observations, sometimes afterward.

You may also be thinking: Where do natural experiments fit in with all of this? Well, they're kind of Method 2.5. The boundaries between these methods aren't always perfectly clear, in any case.

So what we've got here is a sort of hierarchy of ways of understanding the world. There's a tradeoff between general applicability and the amount of knowledge you get. Experiments rarely work, but when they do you get a lot of understanding. History works any time, but you rarely understand why things happen. Empirics is (are?) in the middle.

But what I see is a lot of people dissing empirics as somehow inferior to experiments. That's what's really behind the "econ isn't a science" trope. Why does this happen? Don't people get that empirics, though less powerful than experiments, can be applied in a much wider range of situations? 

My guess is that it's all because empirics came out of order. History is cheap, and experiments are also (sometimes) very cheap - think of Mendel growing peas in his garden. But empirics usually requires Big Data, which is expensive. And even the simplest empirics requires statistics. So while we got written history over 8000 years ago, and experiments almost 1000 years ago, we didn't get modern statistical empirical methods until maybe 100 or 200 years ago. And only recently, with the rise in information technology, have empirics really exploded.

To a lot of people, the empirics revolution must seem like a step backward. We look back to the huge successes of chemistry and physics and medicine in the last few centuries, and the rock-solid theories they generated, and we compare it to the regressions economists are running nowadays, and we say "Ugh, this isn't science!" We look at the progression from history to experiment, and we think that new methods (if they exist) should go the same way - i.e., they should lead us to deeper understanding. But empirics, instead, goes in the direction of wider applicability with less-deep understanding, and that rankles some people.

I don't think they should be rankled. Empirics is an innovation that allows us to know some things about big phenomena that previously we could only understand through written history. It's not a substitute for experiments, it's a complement. It's a valuable addition to humanity's toolkit, whether you want to call it "science" or not.


  1. Gorby's characterization is within the "inductivist" tradition, which is fundamentally wrong. And so the discussion of it is also off. Read Popper's Conjectures and Refutations. It is not obvious how to apply Popper to social science, and Popper himself didn't do such a hot job. But if you start with Popper at least you will have a more sound view of "hard" science. Isabelle Tsakok's Success in Agricultural Transformation actually does Popperian economics, meaning using the same standard of testability and rigor in testing.

    1. No, Popper got hard science seriously wrong as well.

  2. Anonymous6:33 PM

    Interesting theory, but if you were right, then these same people who don't think Economics is a science would also think the same thing about other observational sciences like astronomy or evolutionary biology. As a former astronomer (sort of...), my experience is that that is most definitely not the case as both fields are generally well respected fields.

    You know what I think? Some "hard science" types just don't think much of the social sciences. Period. And they look for reasons/arguments to back up their dislike. I know this because I used to be one of those people, before I broke down and tried Econ (and haven't looked back since).

    1. "You know what I think? Some "hard science" types just don't think much of the social sciences. Period. And they look for reasons/arguments to back up their dislike. I know this because I used to be one of those people, before I broke down and tried Econ (and haven't looked back since)."

      Most of them seem to be lab types, who don't understand that they're in a special, privileged position.

  3. From an evolutionary viewpoint history is the basic adaptive form of knowledge. History can be embedded in the evolved structures of the organism. For example, we have evolved eyes because looking at stuff is very useful. More sophisticated organisms are able to record their own histories and use this in combination with their evolved structural knowledge to make smarter decisions.

    Humans are able to package historical knowledge into transmissible stories: talk, writing and later on youtube videos. This process requires that the stories must be intelligible and memorable. We intuitively select stories on this feels-good basis. This might be called truthiness. The adaptive feature is that the stories are useful. Usefulness can be determined by personal experience, by some kind of group choice, or, failing that, by the eventual survival of the holders of the view.

    Actually being true is related to usefulness and truthiness but the relationship is extremely messy. Lots of wacky beliefs work. This is particularly true for social animals where many of the useful stories are about hallucinations like values, conventions, reputations, money, fashion, gods, and so on.

    Doing science is a radical new activity from a biological perspective. It can really only appears in times of surplus when the problem of finding breakfast is solved and days can be spent in the lab exhaustively testing possibilities. That is why, for example, global warming deniers resort to anecdotes: science is unnatural to us, history is built in. From a psychological perspective, science requires faith.

    Empirics is even weirder than science. At the end of a scientific enquiry we at least have a believable story with a degree of truthiness that people can relate to. (Hopefully it also has abstract quality of being true too, but determining this requires special skills.) Empirics doesn't even produce a good story. After all the number crunching it comes down to "Computer says X." The calculations have too many inputs, too many steps, and too much abstract theoretical basis to be intelligible to most people. The result may be incredibly useful but it still feels like black magic.

  4. Anonymous9:03 PM

    Excellent Noah. People who say this are usually stupid and I don't think I am, but you are one smart dude!

  5. My takeaway is that Noah Smith is advocating the bombing of Mongolia before it is too late.

  6. Ozimek rejects use of the scientific method and controlled experiments as a criterion for science. He apparently gets to redefine the word 'science'.

    Feynman's definition: "Science is the belief in the ignorance of experts."

  7. If history were a science we would have "solved" many questions of the past by now. But there will always be just one more book about Henry iV or Abraham Lincoln. In describing the past we describe the present.

    Similarly there's a feedback loop in economics. If we take self interest for granted then we increase our tendency to self interest. If we assume self interest is crude or vulgar we mitigate against it. Georges Lefebvre notes that the majority of the aristocracy were not rich and did not no how, or want to become rich. They wanted their privileges but not more. This has been lost on defenders of contemporary economic "science".

    There's no reinforcement in geology.

    Economics is the new vulgar Marxism.

    1. http://press.princeton.edu/titles/8032.html

  8. Well if economics built airplanes and made medicines it'll get much more respect from those guys. I like to think that much of the social progress is thanks to the ideas of dead economists, but will never be convincing for most of the critics.

    Anyway I don't see many of the same people doubting the science of Astronomy or theories like plate tectonics or evolution that are also empirically confirmed.

    1. bjdubbs2:53 AM

      If economics built airplanes or developed vaccines, it would be engineering, not science. I love how Gobry comes along and says, "nobody knows what science is, but I'll tell you - it's engineering!" And then he rehashes the most boring and long-refuted versions of Aristotle and Bacon as if he just got back from the archives.

  9. Anonymous4:21 AM

    Actually practically all of astronomy and cosmology have almost the same problems as economics. That is true for a lot of natural sciences (most of ecology, climatology ...). Basically he makes the point that all of the string theory is as far from the science as possible (with this one, I might even agree :) )...

  10. Alex Bollinger5:14 AM

    Yes. I also don't see how Gobry's critique couldn't be applied to biology; how dare those people who tag and follow seals or those others who collect and analyze thousands of DNA samples from plants call themselves scientists!

  11. Isn't the distinction between history and empirics that history (at least, since von Ranke) proceeds by using different sources on the same event, because the differences are informative, while empirics compares standardised sources on multiple events?

  12. Anonymous7:14 AM

    I think "History" is more powerful than you think. Simply by selecting one out of an infinite array of facts you are assigning importance to that fact.
    Two facts about Mongolia in the 13th century:
    "Ghengis Khan invaded North China"
    "There were a lot of earthworms"
    Only one is seen as historically important.

    By assigning importance to that fact you are imposing an interpretation on the world. This becomes more important if you construct a chronology i.e. put the facts in time order since this links important facts together as being jointly important.

    All this is before you start to compare and contrast facts to work out motivations, the environment in which decisions were made etc.

    You can get a long way by using "History".

    1. Actually, you might not be too far off, I believe there is some geological evidence indicating that the Steppes were very wet during those years. http://www.scientificamerican.com/article/rise-of-genghis-khan-linked-to-unusual-rains-in-mongolia/

  13. This makes sense Noah.

    Minor quibble: I think the difference between 1 and 2 is more a matter of degree. Plus, one of the advantages of 1 is that it sometimes lets you tease out causality. Did Columbus sail to America to find the gold that would be needed to finance the coming inflation? Nope. that is historically very implausible. Gold influx caused inflation, not vice versa. That's why Friedman did history.

  14. It seems obvious that Economics is Computer Science, but that computers just haven't gotten strong enough yet.

    First remove currency, and move to only digital currency. Go to block chain if you want.

    Next put a unique ID scannable on every non-consumable asset, and store a transaction history stronger than CarFax or Home title.

    Next store records of all transactions for services but also broadcast all of them in real time.

    Now track, as apps easily do, when people work, play, rest, etc.

    There are only 9B people.

    Move all govt services into a open source cloud based platform, same code used by all govts.

    How is economics now not a science?

  15. Noal,

    A good post. However, you miss out the most critical ingredient to make ECON to a true science. Alchemy has history, empirical data, and controlled experiments. Alchemy is not a science. It is the laws of quantum physics that turns alchemy into modern chemistry, which becomes a branch of physics.

    The question whether economics is a science should be considered as a settled issue.

    Economics does have immutable laws: physics laws of social science. Please check out

    Economics does have a universal mathematical framework like Maxwell’s Equations for
    electromagnetism: a fundamental equation of economics. Please check out

  16. I was reading along thinking this is all quite reasonable. and then i hit

    "we got written history over 8000 years ago"


    try less than five. and that's if you're counting very basic historical information such as the early egyptian king lists. the first detailed historical text in which more than half of what was recorded actually plausibly happened was about 400bc (thukydides).

    1. Bill Ellis11:11 AM

      Your Western bias is showing.... "Chinese archaeologists studying ancient rock carvings say they have evidence that modern Chinese script is thousands of years older than previously thought.
      State media say researchers identified more than 2,000 pictorial symbols dating back 8,000 years, on cliff faces in the north-west of the country. "


      But heck, I think an argument can be made that recording history is much older...The Cave Paintings in Lascaux are about 17,000 years old and it could be (it has been argued) that they contain information important to the people of the time.

      It just seem natural to me that humans were probably finding ways to record important historical information through a system of symbols from the beginning. History important to hunter gatherers could have been recorded in the form of an arrangement of beads or patterns on utensils and tools.

  17. Demarcating science from non-science is challenging. Few, if any, objectives, practices, or deliverables feature in every science, while failing to feature in every non-science. Still, we know that (for example) quantum mechanics is science, while a Thomas Friedman column is not science. "Science" may be a family resemblance term, but it is difficult to determine precisely which family resemblances come into play. There is a large literature in the philosophy of science on this subject, but it succeeds only in refuting widely held conceptions of science (including those held by working scientists), while failing to advance a compelling alternative.

    Whatever the proper conception of science is, clearly economics is a borderline case. Even if we ignore critiques of the scientific status economics by people who know little about either economics or science, there remain compelling critiques of the objectives, practices, and deliverables of economics qua science, along with compelling defenses of these same things. (One who fails to find compelling arguments on one or the other side is not looking hard enough.)

    If economics is a borderline case, the dispute of whether it is a science will go on indefinitely without resolving the more important question of whether economics produces understanding of the economic realm. Since the economics-is-science view primarily serves to support this contention, while the economics-is-not-science views serves to undermine it, we ought to cut out the middleman and stop worrying about whether economics is or is not a science.

  18. OK, so Noah you mentioned experiments but did not point out that economists (along with psychologists) have been doing controlled lab experiments for quite some time now, along with more recently field experiments. One must admit these are not as controlled as some in chemistry and physics, although arguably better than some in such fields as ecology and climatology. It has gotten to the point where experiments are now being put together with mechanism design to set up real world institutions such as auctions.

    I think there is also some break here between micro and macro. Much of the complaint about econ not as science has to do with its failure to forecast the financial collapse, and so on. This is macro, which we all know is a mess. In many micro situations, the abillity to make forecasts based on theoretically founded models that have been empirically and even experimentally validated is much stronger, if still not at chemistry levels.

    Barkley Rosser

    1. I actually do experiments! So sure, I know about that. But it's still a relatively small % of econ papers - less than 10%.

      And sure, macro necessarily involves more history than micro, despite people pretending that the two are equally subject to statistical analysis.

    2. Bill Ellis9:07 PM

      Are there any papers on your experiment s available to the public and written at a level a layperson could understand?
      What did you try and test?

  19. Anonymous3:34 PM

    Of course there's a science of economics. It's just not clear how many economists are doing science, rather than some math-heavy school of rhetoric.

    1. Well, there does seem to be quite a bit of that around, still...

  20. Economics is a mess. Now, you can go out of your way and call the mess science. You can solve the problem by terminating the subject.


  21. Anonymous5:49 PM

    I'm not sure if someone else said anything about it but I think that empirics without a theory framing (some of the) observations is an autistic chimera. science is fueled by purpose and purpose at some point (fairly early on) is about sorting out what is important and what's not.

  22. Hi Noah (and anyone who has a view), I always wondered how Astronomy fits into the science framework. U can't reproduce the Big Bang. So Astronomy is like history?

    Also, I think scientists need to take the post-modernist critique seriously. I tend to use the term "critical realism" to describe how I think about reality. See: http://en.m.wikipedia.org/wiki/Theological_critical_realism

    We need some degree of self-doubt in our ability to "know" truths.

    Cheers, Kien

  23. I'll say again: there's no feedback loop in geology.

    The "objective" study of human behavior affects human behavior.
    A commenter at Crooked Timber asked "where is my Asimovian psychohistory".

    The culture of the web self-selects for fans of science fiction, technocrats and technobrats who can't imagine their own priors.

  24. Anonymous4:08 AM

    This is off of wiki:

    "Science (from Latin scientia, meaning "knowledge"[1]) is a systematic enterprise that builds and organizes knowledge in the form of testable explanations and predictions about the universe.[2][3][4] In an older and closely related meaning, "science" also refers to a body of knowledge itself, of the type that can be rationally explained and reliably applied. A practitioner of science is known as a scientist."

    Much more inclusive than his definition. Maybe econometrics doesn't use perfect random sampling, maybe it doesn't use controlled experiments, but it's definitely systematic, and useful for testing explanations and predictions.

    I haven't read his article, but the fact he used the financial crisis just seems ridiculous. 1) if I told him he would die tomorrow, and I had a very accurate prediction, what would happen? Would he change his behavior in a way that would make the prediction useless? 2) What about flipping a coin? What if a financial crisis was dependent on him correctly guessing? Apparently science is harder when you have atoms that have feelings, non-deterministic processes (obviously there are some stochastic processes in the hard sciences, but the point is the same), and the impossibility of most controlled experiments. I could demarcate science into two groups, sissy sciences that have all of these benefits, and the difficult sciences that don't. I wouldn't though, because honestly all professions are equalized out by difficulty. Which part of the triathlon is hardest? Trick question, the distances are set so that they're even (give or take). Biking is mathematics, easy to do on your own time without experiments. It's easier to get farther, so you can go farther. Some of the hard sciences are running, which can't get as far due to the obvious limitations that are inherent to it compared to math. Economics is swimming. Many more inherent problems, so it's much more difficult to get farther. Each profession tries to look at how far the other got and use it as justification that their field is better, so it's like someone biking making fun of someone swimming for not going as far. Surely, working just as hard and getting farther must mean you're better....right?

  25. No controlled experimentation?
    So much for astronomy as a science.
    [Insert abusive characterization of P-E Gobry’s intellect here.]