Sunday, November 01, 2015

Robert Lucas in biology class


Back in August, a bunch of people were talking about Paul Romer and Bob Lucas and history of macro and stuff like that. Somehow I missed this post, where Brad DeLong dug up a Bob Lucas memoir and made fun of Lucas' college biology class exploits. For reference, here's a longer version of Lucas' story:
The only science course I took in college was Natural Sciences II - a biology course. We read a modern anatomy text, and also selections from Darwin, Mendel, and others... 
[T]here was nothing spooky about Mendel’s genetic theories. They were clear, they made some kind of sense (though there was nothing molecular in our Nat Sci II readings), you could work out predictions that would surprise you, and these predictions matched interesting facts. We did a classroom experiment with fruit flies, focused on eyes, and pooled the results. Our assignment was to write up the results in a lab report and compare them to predictions from a Mendelian model. I had not enjoyed the actual lab work but I liked writing the report and spent the better part of my weekend on it. It was the first time I can recall ever working out the predictions of a scientific theory from its basic principles and testing these predictions against experimental evidence. 
On Sunday evening, my friend Mike Schilder asked to copy [my report on the fruit fly experiment]. I agreed...Mike came back in half an hour, and told me: “This is a good report, but you forgot about crossing-over.” “Crossing over” was a term introduced to us to describe a discrepancy between Mendelian theory and certain observations. No doubt there is some underlying biology behind it, but for us it was presented as just a fudge-factor, a label for our ignorance. I was entranced with Mendel’s clean logic, and did not want to see it cluttered up with seemingly arbitrary fudge-factors. “Crossing over is b---s---,” I told Mike. In fact, though, there was a big discrepancy between the Mendelian prediction without crossing over and the proportions we observed in our classroom data, too big to pass over without comment. My report included a long section on experimental error, describing the chaotic scene that generated the data and arguing that errors could have been large enough to reconcile theory and fact. I handed it in as written. Mike, on the other hand, took my report as it stood, except that he replaced my experimental error section with a discussion of crossing over. His report came back with an A. Mine got a C-, with the instructor’s comment: “This is a good report, but you forgot about crossing-over.” 
I don’t think there is anyone who knows me or my work as a mature scientist who would not recognize me in this story. The construction of theoretical models is our way to bring order to the way we think about the world, but the process necessarily involves ignoring some evidence or alternative theories - setting them aside. That can be hard to do - facts are facts - and sometimes my unconscious mind carries out the abstraction for me: I simply fail to see some of the data or some alternative theory. This failing can be costly and embarrassing to me, but I don’t think it has any effect on the advance of knowledge. Others will see the blind spot, as Mike did with crossing-over, keep what is good and correct what is not.
DeLong makes fun of Lucas for rejecting chromosomal crossover. which is indeed a real thing, and the discovery of which won a Nobel in 1933. It does seem kind of lazy, actually. Even before Wikipedia, it wouldn't have been hard to go grab an advanced textbook and look up how chromosomal crossover works. Lucas is unhappy that it's presented as a fudge-factor, but by the time you're an undergrad you should be too old to depend on the teacher for 100% of your knowledge. If something isn't adequately explained to you, go look up how it works! 

Lucas says that this episode demonstrates a professional weakness of his - the tendency to want to over-simplify theory in order to "bring order" to the world. But I think it demonstrates something slightly different and more worrying: selective empiricism.

In his bio class, Lucas did an experiment on fruit fly inheritance. After the results didn't completely agree with the predictions of basic Mendelian theory, he attributed the discrepancies to experimental error - basically, to measurement noise. Fine (if slightly lazy). But then he takes the experimental result as support for the Mendelian theory, despite the presence of all that experimental error!

If the experimental situation was such a "chaotic scene," then any seeming agreement between Mendelian theory and the lab results might well have been an experimental error. So if college-age Lucas had really been an empiricist, he would have said "This experiment was such a chaotic scene that it provides only very weak support for Mendelian theory." Instead, he concludes that the experimental setup was reliable enough to support the theory that makes "some kind of sense" to him, but too unreliable to indicate the presence of additional phenomena like chromosomal crossover.

In other words, Lucas' conclusion from the experiment relied strongly on his own priors. Or if you prefer a frequentist term, he protected the null hypothesis. That has little to do with oversimplification; it's just a manifestation of confirmation bias. You pick the theories that make sense to you, and believe in them until the data decisively refute them.

But hey, who among us didn't have silly ideas in college?

16 comments:

  1. Anonymous2:22 PM

    Noah, I enjoy reading your blog, but I you often read you attempting to apply a Bayesian rationale for one thing or the other where you focus on the "prior" -- in the this case, Lucas's assuming that his data supported a Mendalian genetic model, when perhaps it didn't or did so very weakly. I just wanted to point out that the more important distinction of being a Bayesian is in its definition of probability, which allows for a mathematical framework for logical reasoning with incomplete or partial information (which is exactly what a scientist does). Practically speaking, this means everything (i.e. an event, parameter, model, etc.) has a probability distribution associated with it, and this is the major philosophical distinction between Frequentism and Bayesianism.

    In this case, Lucas likely started with a uniform prior (whether he realized it or not) applied Mendal's law to his data, then determined his experimental results didn't match the predictions that well. A Bayesian's conclusion would be that the posterior probability distribution on some form of Mendal's law (i.e. presumable the model Lucas used) is weak, given Lucas's experimental data. If Lucas had reason to apply a strong prior in support of Mendal's Law, he should have explicitly stated it and applied it to his application of Bayes theorem. (Of course, I wouldn't actually expect a freshman undergraduate of biology of Lucas's era to do this, just trying to make a point.)

    What it sounds like Lucas actually did, however, was realize that his data poorly matched his model, and tried to explain the discrepancy (i.e. the broad posterior distribution for the model) with uncertainty in the model parameters. His data could have been so poor that any other model (crossing over effect or not) would have resulted in a similar or better posterior distribution, again given Lucas's data. This might have shown that perhaps his model was not so good after all. And again, if Lucas had a strong prior, he should have stated it clearly (i.e. mathematically) and applied it in the Bayesian computational framework (i.e. Bayes theorem). But he didn't, so what he did in essence was just bullshit why his model is actually correct -- i.e. ignore that his posterior distribution on his model was really broad, which should have prompted concern in the model.

    Again, I definitely get your point on the concept of Lucas's "prior", and I'm sorry to harp on this subtle point, but calling Lucas's leap of faith in his model a "prior" kind of maligns Bayesianism. A Bayesian clearly states the prior, they don't use it as an excuse to justify anything they want -- that's just classic "hand waving after the fact."
    In case anyone is interested, a very good free introduction reference to Bayesiansim can be found by d"Agostini here:
    http://arxiv.org/pdf/physics/0304102.pdf
    Again, I don't mean to come off as overly critical of either you or Lucas, nor to say I'm an expert representing Bayesianism, but I do have trouble with the casual use of "prior" defining Bayesianism, which I believe to be a popular sentiment. It's the Bayesian definition of probability and how it's used that's most important, in my opinion. The prior is just an important tool to explicitly state one's assumptions and prejudices (clearly and beforehand).
    But I'm still a student on the subject, so open to counter criticism.

    -M

    ReplyDelete
  2. Isn't it a wonder that this kind of thinking dominated Econ for 30 years following the Lucas Critique? Must've done a lot of good.

    ReplyDelete
  3. Anonymous6:35 PM

    "...you forgot about crossing-over."

    Arising as it has from the Lucasian folklore , that phrase would be fitting as shorthand on twitter and such for the critique of economists who always manage to "forget" about distribution ( of incomes , wealth , debt , all of the above ).

    ReplyDelete
  4. Anonymous11:08 AM

    I agree with anonymous number 1.

    You don't know statistics that well, so your analogies are terrible, not only for the bayesian ones.
    Take for instance your claim: "Or if you prefer a frequentist term, he protected the null hypothesis."

    The usual null hypothesis here would be that random variation explains the observations, so to protect the null here would mean to claim that you can't reject the hypothesis the variations came from random noise.



    ReplyDelete
    Replies
    1. This comment has been removed by the author.

      Delete
    2. Anonymous3:41 PM

      This is Anonymous 1, and I would like to disassociate myself from this commenter. I have criticisms with your (Noah's) use of the word "prior," but would hardly suggest you don't have an understanding of statistics. That's just silly. (Hell, I'm more comfortable suggesting I don't understand statistics!) Again, I do very much like your blog.

      But for anybody who's interested (you too, Noah), read d'Agostini's free summary paper . It's a very digestable Bayesian primer -- no E. T. Jaynes, of course -- but it might clear up the occasional Bayesian / Frequentist discussions found on this blog. I believe there definitely is a "there" there, and it's not the prior.

      Delete
  5. DeLong digs up a 14-year-old memoir of Lucas's, tries to make him look bad, and you pile on with accusations of laziness and silliness. What a cheap shot.

    ReplyDelete
    Replies
    1. We're obviously just kidding. At least, I am. I mean, it's a memoir about a college class.

      Delete
    2. "...it's a memoir about a college class."

      Exactly. So who cares? Why write about it? If I were Lucas, I wouldn't think, oh ha ha, Noah just made a funny joke.

      Delete
    3. You know, it's kind of strange that in person you are constantly laughing at everything, but on the internet you take everything seriously! I mean, come on!

      Delete
    4. And come on. If you can't make fun of the most influential macroeconomist of all time for something he said in a college class, what *can* you make fun of? ;-)

      Delete
  6. Anonymous2:45 PM

    Lucas' priors are steeped in political and social ideology - he does his best to conceal them in the cloak of scientificism.


    Henry.

    ReplyDelete
    Replies
    1. Anonymous10:29 PM

      Prejudice is not a prior. A prior needs to be stated and quantified, otherwise you're just making shit up to justify whatever you want.

      Delete
    2. Anonymous2:32 AM

      "you're just making shit up to justify whatever you want."

      Yup!


      "A prior needs to be stated and quantified"

      The probability that Lucas' ideological predilections inform his economic theorizing is 1.

      H.

      Delete
  7. Brad's post caused my respect for Lucas to increase enormously (no I never expected to type that either).

    As Lucas noted, the history of macroeconomic thought is illuminated by this early and very clear case of his tendency to have a strong prior or to protect the null or to use the correct technical term derp (hey if you don't use the word as you defined it then who will). I think his assertion that this has lead only to personal embarrassment is incorrect -- he is so influential that his derp infected the profession (I think he was so influential in part because he was so derptastic -- with little data a lot of derp can go a long way).

    But to me the main point is that his willingness to tell the sad story demonstrates intellectual integrity. I fear he doesn't understand just how humiliating it should be (I have a BA and an MA in biology and also would still be humiliated if I had made such a mistake in high school). But I am still very favorably impressed. Yes few among us didn't have silly ideas in college. More importantly, even fewer among us are willing to tell the world about the silly ideas we had in college.

    On the other hand, I have an even lower opinion of the Macroeconomics profession. Now I know that we were influenced not only by someone blinded by derp, but by someone who has been willing to frankly admit that he is blinded by derp.

    And I don't even want to think of helpless fruit flies which have fallen into the hands of Ed Prescott.

    By the way, Lucas's null was not Mendelian. Mendel was wrong too (back in 1870 and right and decades ahead of everyone else). But that's because Mendel thought that genes were unlinked so the chance of a crossover between any two genes is 0.5 (even if they are on the same chromosome). The new insight (discovered wtih fruit flies by Morgan, Morgan and Sturtevant) is that genes have something to do with chromosomes and, if two genes are reasonably close to each other on the same chromosome, then the chance of a crossover between them is markedly less than 50% .

    ReplyDelete