Wednesday, December 14, 2016

Academic signaling and the post-truth world


Lots of people are freaking out about the "post-truth world" and the "war on science". People are blaming Trump, but I think Trump is just a symptom. 

For one thing, rising distrust of science long predates the current political climate; conservative rejection of climate science is a decades-old phenomenon. It's natural for people to want to disbelieve scientific results that would lead to them making less money. And there's always a tribal element to the arguments over how to use scientific results; conservatives accurately perceive that people who hate capitalism tend to over-emphasize scientific results that imply capitalism is fundamentally destructive.

But I think things are worse now than before. The right's distrust of science has reached knee-jerk levels. And on the left, more seem willing to embrace things like anti-vax, and to be overly skeptical of scientific results saying GMOs are safe. 

Why is this happening? Well, tribalism has gotten more severe in America, for whatever reason, and tribal reality and cultural cognition are powerful forces. But I also wonder whether a few of science's wounds might be self-inflicted. The incentives for academic researchers seem like they encourage a large volume of well-publicized spurious results. 

The U.S. university system rewards professors who have done prestigious research in the past. That is what gets you tenure. That is what gets you a high salary. That is what gets you the ability to choose what city you want to work in. Exactly why the system rewards this is not quite clear, but it seems likely that some kind of signaling process is involved - profs with prestigious research records bring more prestige to the universities where they work, which helps increase undergrad demand for education there, etc. 

But for whatever reason, this is the incentive: Do prestigious research. That's the incentive not just at the top of the distribution, but for every top-200 school throughout the nation. And volume is rewarded too. So what we have is tens of thousands of academics throughout the nation all trying to publish, publish, publish. 

As the U.S. population expands, the number of undergraduates expands. Given roughly constant productivity in teaching, this means that the number of professors must expand. Which means there is an ever-increasing army of people out there trying to find and report interesting results. 

But there's no guarantee that the supply of interesting results is infinite. In some fields (currently, materials science and neuroscience), there might be plenty to find, but elsewhere (particle physics, monetary theory) the low-hanging fruit might be picked for now. If there are diminishing returns to overall research labor input at any point in time - and history suggests there are - then this means the standards for publishable results must fall, or America will be unable to provide research professors to teach all of its undergrads.

This might be why we have a replication crisis in psychology (and a quieter replication crisis in medicine, and a replication crisis in empirical economics that no one has even noticed yet). It might be why nutrition science changes its recommendations every few months. It might be a big reason for p-hacking, data mining, and specification search. It might be a reason for the proliferation of untestable theories in high-energy physics, finance, macroeconomics, and elsewhere. And it might be a reason for the flood of banal, jargon-drenched unoriginal work in the humanities.

Almost every graduate student and assistant professor I talk to complains about the amount of bullshit that gets published and popularized in their field. Part of this is the healthy skepticism of science, and part is youthful idealism coming into conflict with messy reality. But part might just be low standards for publication and popularization. 

Now, that's in addition to the incentive to get research funding. Corporate sponsorship of research can obviously bias results. And competition for increasingly scarce grant money gives scientists every incentive to oversell their results to granting agencies. Popularization of research in the media, including overstatement of results, probably helps a lot with that.  

I recall John Cochrane once shrugging at bad macro models, saying something like "Well, assistant profs need to publish." OK, but what's the impact of that on public trust in science? The public knows that a lot of psych research is B.S. They know not to trust the latest nutrition advice. They know macroeconomics basically doesn't work at all. They know the effectiveness of many pharmaceuticals has been oversold. These things have little to do with the tribal warfare between liberals and conservatives, but I bet they contribute a bit to the erosion of trust in science. 

Of course, the media (including yours truly) plays a part in this. I try to impose some quality filters by checking the methodologies of the papers I report on. I'd say I toss out about 25% of my articles because I think a paper's methodology was B.S. And even for the ones I report on, I try to mention important caveats and potential methodological weaknesses. But this is an uphill battle. If a thousand unreliable results come my way, I'm going to end up treating a few hundred of them as real.

So if America's professors are really being incentivized to crank out crap, what's the solution? The obvious move is to decouple research from teaching and limit the number of tenured research professorships nationwide. This is already being done to some extent, as universities rely more on lecturers to teach their classes, but maybe it could be accelerated. Another option is to use MOOCs and other online options to allow one professor to teach many more undergrads. 

Many people have bemoaned both of these developments, but by limiting the number of profs, they might help raise standards for what qualifies as a research finding. That won't fully restore public trust in science - political tribalism is way too powerful a force - but it might help slow it's erosion. 

Or maybe I'm completely imagining this, and academic papers are no more full of B.S. than they ever were, and it's all just tribalism and excessive media hype. I'm not sure. But it's a thought, anyway.

37 comments:

  1. You think prestige is about getting students? I would have thought getting donations and/or grants was a bigger motivation.

    ReplyDelete
    Replies
    1. Think you have it backwards there

      Delete
  2. I am fairly sure almost none of the public read those papers and few even read news about them, but that those who find them favorable to their agendas promote them. Trump as BS artist likes to give his audience what they want. Whether he can lead them to the promised land or will be content to have them believe they are already there is anyone's guess.

    ReplyDelete
    Replies
    1. GrueBleen7:12 PM

      I'm quite sure that none of "the public" read those papers, and I'd also agree that relatively few read news about them. But that is now with, as Noah covers, a veritable flood of papers across a very wide range of fields from all around the world. And, unfortunately, a much diminished science commentary capability.

      Once upon a time there were reputable science commentary publications - but New Scientist, just for one example, hasn't been readable for years. And as for the average science commentary in the MSM, it's pitiful in general (with just a few prominent exceptions). Just consider the commentary provided by, and about, for instance, Susan Greenfield or Ray Kurzweil.

      But once upon a time, popular science had quite a following - just consider all those 'Popular this-that-n-tother' magazines that you hardly see any more (though Popular Science still seems to be alive). And the public response to the likes of Einstein and Feynman. And to the public response to Carl Sagan's 'Cosmos' and the very many Attenborough series.

      What we seem to be seeing now is the 'memes of ignorance' spreading and becoming tropes that end up being defining components of 'identity politics' which indeed lead us straight on into "post-truth".

      What I think we really have, is an abysmal failure of education from the earliest years onwards. Most grown adults are barely even numerate, much less scientific. Adapting a quote I once encountered about mathematics, it's that "For those going on to related careers, school science teaching is too little, too late. For everybody else, it's too much, too soon" (and hence makes zero impact).

      Delete
  3. Another solution would be for university positions and research grants to stop being given out based on prior research results.

    Right now there are millions of dollars on the line for researchers trying to produce results. It is the difference between a well paid career or a life of destitution while being a slave to huge student debt. It's no wonder a lot of students are blind to the flaws in their research.

    I believe strongly that teaching positions and research grants should be given out based on criteria that are only incidental to research results.

    Evaluate profs and grants based on:

    1. Domain knowledge (test the applicants) 2. Math skills (test the applicants) 3. Motivation and leadership 4. Prior and current research proposals (but ignore results, especially the fact that they were published or not). 5. Other skills such as written and oral communication.

    Universities should not rely on journals to evaluate their professors. This corrupts the whole system. Journals have different goals. They want to publish good research with interesting results. Universities should hire smart researchers that perform good research with interesting _questions_ regardless of the results.

    If universities keep giving out jobs based on having generated interesting results, they are going to keep getting researchers that ignore biases and publish whatever results are interesting whether they are true or not.

    This is also a vicious circle. When you are in academia and you see all these people using poor methodology and ignoring obvious biases getting rewarded and advancing faster in their careers you likely either leave academia or start doing the same. Those that stay often teach their grad students to use the poor methods. At some point some universities might only have fake researchers left.

    ReplyDelete
  4. Anonymous5:59 PM

    There is another solution. Value research on its real world impact rather than just rewarding theoretical calisthenics.

    -Asssistant Prof.

    ReplyDelete
    Replies
    1. Even accepted the premise (that real-world impact is the only source of value for research), the real-world impact of research is often only up for assessment on very long time horizons. Physics is a good example of this, philosophy an even better one (he said, poking the hornet's nest ungingerly).

      Delete
  5. So same number of academics competing over fewer jobs, but the assessment system remains unchanged. And you think this will lead to less BS?

    In order to fix a lemon market, you need to improve certification, not reduce the number of buyers.

    Now we may want to spend less on research, and this may be because there are diminishing returns, and this may lead us to unbundle research from education. But this would exacerbate the BS problem, not reduce it.

    ReplyDelete
  6. All the low hanging fruit is gone so we need less scientists? Seems backwards to me, unless you just want to give up on technical progess. Harder science will require more scientist collaborating, the low hanging fruit are things that could be done by individuals. Should follow model of LIGO and CERN. Big groups of scientists in well designed collaborations can do unimaginably difficult things and then they can share credit.

    ReplyDelete
  7. Anonymous9:05 PM

    I think this analysis misses something big- the huge explosion of contingent/part time academic labor. Most universities aren't teaching undergrads with research professors they are teaching them with adjuncts and lecturers.

    The result is that research is being done by postdocs and grad students who no longer have faculty research roles to move into later, and the competition for research positions has intensified. The result of more pressure is more hype. That psych experiment that didn't pan out? Mine it for a weird comparison with a good p-value and turn it into a TED talk,etc.

    When my advisor was in grad school, one or two decent papers and a well known advisor was enough to get a faculty position at a good university. Now, in my field, you need 5-6 years of postdocs and well received papers in a multiple adjacent sub-fields to have any shot at an interview.

    ReplyDelete
    Replies
    1. This seems totally right to me. What Noah is ignoring is that adjunctification of the academy is itself a driving force in the explosion of published research.

      It's also interesting that Noah tied tenure with research. Much of this pressure would be alleviated if colleges and universities, like primary and secondary schools, offered tenure (and competitive pay, natch) for teaching-only positions. You'd get scores of graduate students deciding that they could be happy with that (maybe it was all they wanted in the first place!) and curtailing their research output and beefing up their teaching experience and quality. There's even a rationale for tenure beyond it being a major employment perk: researchers say controversial things, but so do teachers (because they must teach controversial views, issues, and theories); they need to be protected from political reprisals too.

      (And, personally, anything that sticks the capitalists and their administrative dogs in the eye is fine by me....)

      Delete
  8. I think you've touched on a lot of issues with academia, but I'm not sure about the narrative you're trying to pull together.

    A couple critiques(no particular order)
    >Exactly why the system rewards this is not quite clear,
    This doesn't seem that strange. In a somewhat naive way, if you want the best, one of the ways to evaluate candidates is based on previous work. Someone who is changing his field is more likely to keep producing groundbreaking research. And same for quantity- if you're going to hire someone, you may as well hire the guy who publishes 3x a year instead of the guy who publishes once. (The signaling thing also matters). It's a silly system, but as far as i know, no one has come up with better, right?

    >Given roughly constant productivity in teaching, this means that the number of professors must expand
    Not sure about this. Productivity has rapidly increased with larger and larger classes sizes, hasn't it? Probably not enough to keep up with demand, but it's nontrivial. Our intro courses these days are ~300-400 students. Honestly the bigger constraint is the lecture hall size (~150-200). Not sure how much more you could scale it before it gets unfeasible, though. There's a quality drop once you past the 40-50 mark, when the professor can't really give 1 on 1. (A lot of which gets pushed onto TAs.But honestly a TA level is probably fine for the largest courses, or an adjunct prof)

    At the graduate level, that 1 on 1 is much more important, but our graduate classes are still quite small, 8-10 students.

    >It might be a reason for the proliferation of untestable theories in high-energy physics
    I can't speak for other fields, but I'm not sure I'd agree in physics. There's definitely a bunch of junk, but I wouldn't say it's any worse than back in the days people talked about the ether. "Market" forces do tend to keep it in check- there's not a whole lot of funding for untestable theories, and the bar for theory is quite a bit higher (we only have 2 theorists here, compared to a dozen+ experimental).
    You could probably cut quite a bit, but there's a bunch of self correction

    >the standards for publishable results must fall,
    This might not be a bad thing, *as long as they're real*. It's probably not terrible if more people can publish than historically, ala the Einsteins/Lord Kelvin's of the world. A line does need to be drawn, but if you want to roll it back, I think one needs a stronger argument on where the line ideally should be.
    For the fields that are churning out junk, it's easy to point that out.It's a lot harder past that though

    >there might be plenty to find, but elsewhere (particle physics, monetary theory) the low-hanging fruit might be picked for now.
    But why do you have to hire more particle physicists? You can just hire more materials science guys. As far as undergrad courses go, they're interchangeable. The only time you really would need a particle physicist is to teach particle/QFT at the grad level

    >it's all just tribalism
    Something i think that's underrated in this post is scientific results just not agreeing with tribalism. Tribalism is getting worse, for sure, but even if it weren't, just the fact that things like climate change/GMO etc go against the grain might be enough to turn public opinion. It didn't hurt that science was "cool" in a good chunk of the 1900s-a bit of fashion luck.

    In some sense, the trust was never there. Similar to how peoples' views on the economy changed so rapidly post-election- a lot of it was just nebulous feelings, not really peoples' views on the economy because of the way the question was asked.

    Overall, I'm still in the media camp. I think you do bring up a bunch of legitimate problems with academia, but honestly, i don't think the public would even notice most of them.

    ReplyDelete
  9. I really like it that you conflate particle Physics and Economics as the "low hanging fruit". Few, if any, would have the insight to see the commonality thread of "untestable theories". Indeed, particle physics research has pretty much gone to the dogs (to use a most unscientific language). It has been bereft of anything new and testable for 3 decades now (yes, there was that Higgs boson, but when was that proposed?. There were many novel and even profound theories (yes, I do like the multiverse one. If only one could devise a test to take it out of the realm of science fiction!). And then there was String Theory, about which the less is said the better (another example of the triumph of pure mathematics over Empirical research? and that's ALL I will say!).

    The parallels with macroeconomics are legion, or so I am told by my betters. Will we ever know if Secular Stagnation is real? is there a test that will not be contaminated by politics? can economists divorce themselves from politics and/or ideology?

    Of course, I have the answers to all of the above, but I'm not telling.....

    ReplyDelete
  10. Anonymous6:29 AM

    Hey Noah, interesting post and something that I tend to agree on. I'm near the end of my PhD in finance (only need to defend), and one of the reasons that I've left academia is that it can seem meaningless from time to time. I'm in a very good department and the people working there publish very well and quite often, but even all those publications in A journals seem like small advances in our knowledge at best. I got into academia for my love of the big ideas in economics, but the day to day business of academia feels very far removed from those big ideas (outside of teaching them). I never considered the impact on how society feels about science, but I think you have an interesting point here.

    ReplyDelete
  11. "This might be why we have a replication crisis in psychology (and a quieter replication crisis in medicine, and a replication crisis in empirical economics that no one has even noticed yet). It might be why nutrition science changes its recommendations every few months. It might be a big reason for p-hacking, data mining, and specification search. It might be a reason for the proliferation of untestable theories in high-energy physics, finance, macroeconomics, and elsewhere. And it might be a reason for the flood of banal, jargon-drenched unoriginal work in the humanities."

    Noah, these issues are rightly important, but I don't think they actually *are* connected to distrust of science. Specifically the fields of science that face the most distrust - climate science, vaccination theory, basic economics, survey design... are generally not the ones afflicted by those replication issues. In turn, the bits of science whose star is rising - sociology, psychology, stupid economics, they end up being what people trust.

    I've just got shared with me an article decrying 'the left's fascination with positivism' and declaring a need to rely more on anecdotes and emotion, picking out Freakonomics as the one thing the author likes because of all the weird unintuitive results it talks about.

    ReplyDelete
    Replies
    1. I'm seconding this. If Noah's theory was correct, we'd see quite a different pattern of disagreement.

      Delete
  12. Economics is not post-truth but pre-truth
    Comment on Noah Smith on ‘Academic signaling and the post-truth world’

    Noah Smith observes “For one thing, rising distrust of science long predates the current political climate; ...”. This is true, the ‘distrust of science’ is literally built into the core of politics because science is the very antithesis of politics. The guiding principle of science is the distinction between true and false. Scientific truth is well-defined as material and formal consistency. Science has NOTHING to do with trust or credibility or belief but with proof, transparency, explicitness, and acceptance of falsification. All this is antithetical to politics.

    The very signature of politics is to give a shit about scientific standards: “As some one has said, it would seem that even the theorems of Euclid would be challenged and doubted if they should be appealed to by one political party as against another.” (Fisher, 1911)

    To say that we live in a post-truth world is utterly misleading. The fact of the matter is that politics has hijacked science and gradually adapted it to its own modus operandi. Much of what parades a science nowadays is what Feynman famously called cargo cult science, that is, the outer form looks like science, but it is not science, and it does not work.#1

    To say that we live in a post-truth world is to insinuate that politics has been successful in its age-old attempt to corrupt science.

    This is true as far as economics is concerned.#2 Economists claim to do science since Adam Smith/Karl Marx. What they in fact have done is cargo cult science or, more specifically, political economics. Political economics is agenda pushing and fundamentally different from theoretical economics.

    The proper definition of theoretical economics is: the science that tries to figure out how the actual economy works. Scientific knowledge takes the form of a theory which satisfies the criteria of material and formal consistency. A theory is the humanly best mental representation of reality.

    Economics is a failed science. The four majors approaches — Walrasianism, Keynesianism, Marxianism, Austrianism — are mutually contradictory and axiomatically false. Political economics has not produced much, if anything, of scientific value in the last 200+ years.

    The actual state of economics is that of a proto-science, that is, the representative economist lives in a pre-truth world. This is why the word sciences has to be deleted from the “Bank of Sweden Prize in Economic Sciences in Memory of Alfred Nobel”. The claim that economics is ‘sciences’ cannot be upheld. It is provable false.

    Theoretical economics (= science) could never emancipate itself from political economics (= agenda pushing). Political economics is fake science. And all agenda pushers from Smith, Ricardo, Marx to Keynes, Hayek, Friedman and onward to Krugman and Varoufakis are fake scientists. The distrust in political economics is fully justified.

    The most urgent task in economics is to implement the separation of politics and science and to throw all political economists out of the scientific community.#3 This is the only way to de-incentivize incompetent scientists to “crank out crap”.

    Egmont Kakarot-Handtke

    #1 For details and references see Wikipedia
    https://en.wikipedia.org/wiki/Cargo_cult_science
    #2 See ‘Economists: the Trumps of science’
    http://axecorg.blogspot.de/2016/11/economists-trumps-of-science.html
    #3 See also cross-references Political economics
    http://axecorg.blogspot.de/2015/11/political-economics-cross-references.html
    and cross-references Incompetence
    http://axecorg.blogspot.de/2015/07/incompetence-cross-references.html

    ReplyDelete
    Replies
    1. Derryl9:02 AM

      Right. And the first step toward a "scientific" economics is to distinguish between money -- which is created by commercial banks as loans; and economic goods, services and assets -- which are produced by working. Economics treats "created money" and "produced value" as if they are two interchangeable words for the same thing. Meanwhile, most critics of economics believe "the government issues the money": a visibly false belief that is directly contradicted by the empirical evidence that governments are trillionaire bond-debtors, not trillionaire money-issuers. Over the past century monetary reformers have clearly described and explained the world's bank-created money system. But when people hear the truth that private banks, not governments, create the money supply of nations, people simply refuse to believe it. And they go on blaming the government and the central bank for monetary inflation and every other kind of financial woe. Monetary ignorance is the root of all kinds of economic confusion and counterproductive monetary, fiscal and economic policies. Money buys ownership of the world. The power to create and allocate money is the power to decide "who" buys the world. The question, "Who should issue the money?" is a political question. You cannot separate politics from economics. Adam Smith imagined a village barter economy as a free market utopia: free of every kind of monopoly power. Such a utopia has never existed. Civilization has always been ruled by power, not by "markets". "Markets" means one power warring against other powers for ownership and control of finance, commerce and industry. Billionaire corporations that are vast collective enterprises -- not independent butchers, bakers and tinsmiths -- populate the real world political economy. So the first step toward a scientific economics is to stop believing in fairy tales and look at empirically visible reality.

      Delete
    2. Anonymous11:44 AM

      Actually, the first step is for you to learn some modern economics. We are not the ones who conflate real and nominal variables. And you can separate normative economics from positive economics, we can tell you what agency would be better at the monetary printing press under what conditions, but we can't and won't tell you what society ought to prefer.

      Apparently you know Egmont, I'm sure you'll have lots to talk about.

      Delete
  13. Doc at the Radar Station7:33 AM

    Wow, this is getting to be a hot topic. Just read this yesterday and it was salient:
    http://slatestarcodex.com/2016/12/12/might-people-on-the-internet-sometimes-lie/

    "And then there’s bad science. I can’t remember where I first saw this, so I can’t give credit, but somebody argued that the problem with non-replicable science isn’t just publication bias or p-hacking. It’s that some people will be sloppy, biased, or just stumble through bad luck upon a seemingly-good methodology that actually produces lots of false positives, and that almost all interesting results will come from these people. They’re the equivalent of Reddit liars – if there are enough of them, then all of the top comments will be theirs, since they’re able to come up with much more interesting stuff than the truth-tellers. In fields where sloppiness is easy, the truth-tellers will be gradually driven out, appearing to be incompetent since they can’t even replicate the most basic findings of the field, let alone advance it in any way. The sloppy people will survive to train the next generation of PhD students, and you’ll end up with a stable equilibrium."

    ReplyDelete
  14. This comment has been removed by the author.

    ReplyDelete
  15. This comment has been removed by the author.

    ReplyDelete
  16. > Corporate sponsorship of research can obviously bias results.

    I think this is just a throw away statement that's wrong. I don't know of any academic institution that allows a veto over the publication of sponsored research results. there might be a patent allowance but publication is not controlled by the sponsor.

    ReplyDelete
  17. The fight between science and rationalism and a belief in objective reality in one corner and the romantic embrace of various forms of mysticism and faith on the other has been going on for at least five hundred years, and probably much longer.

    "It is difficult to get a man to understand something, when his salary depends upon his not understanding it!"
    -Upton Sinclair

    ReplyDelete
  18. You can always find a lawyer willing to defend the devil for a fee.

    ReplyDelete
  19. I want to nitpick one thing from the introductory part of the post about the "left-wing" bit of anti-vaccinationism. The available polling data suggests that there is actually a slight rightwing bias regarding anti-vaccination:

    1) http://fivethirtyeight.com/datalab/americans-and-scientists-agree-more-on-vaccines-than-on-other-hot-button-issues/
    2) https://mikethemadbiologist.com/2013/08/15/anti-vaccination-has-a-slight-rightwing-bias-and-the-whole-foods-liberal/
    3) https://mikethemadbiologist.com/2014/01/29/vaccination-still-has-a-slight-right-wing-bias-the-dan-kahan-edition/

    Regarding the post topic, in biology, two major contributors are:
    1) research funding doesn't allow for adequate sample sizes (you get what you don't pay for).
    2) many subdisciplines make it difficult to replicate data (e.g., limited data sharing, etc.)

    ReplyDelete
    Replies
    1. Yeah. Worth recalling that Andrew Wakefield endorsed Trump.

      I think GMOs are the biggest leftist science blind spot, but even there I think there's legitimate reasons to be concerned - not so much inherent biological risks but the degree of corporate control and the potential to abuse that.

      Delete
  20. There is a lot to say on this. Here's some:

    A huge thing in research is the misprioritization of quality, deep quality especially versus quantity and speed. From what I've seen, there is much too much emphasis on speed and quantity instead of, and often very much at the expense of, quality – deep intuition and quality of work.

    And a huge profound part of this is the terrible overemphasis on "objective" simple-minded numerical measures at the expense of subjective. In reality, the use of no measure is objective, and therefore beyond reproach, because there is always subjectivity in the choice of the metric. I could grade my students 100% based on their heights, with the top 20% getting A's, and say See! See! It's fair! It's 100% objective!

    Now, of course, there are two huge problems, (1) The choice of that "objective" measure was subjective; and ginormously, (2) It's horrible at reflecting what you truly want to measure and reward, mastery of the course material. If I instead assigned research papers and essay questions that I graded completely based on my analysis of how well they showed correct understanding of the course material, people could scream that it's subjective! It's subjective! But no more so than the typical choices of "objective" criteria, and it would be ridiculously more accurate in reflecting mastery of the course material. But, of course, I should defend my grading with reasoning and evidence.

    With regard to research, we might do far better by having tenure evaluated based on a study of work in progress, and on five years of constant talking to the other faculty and working with them, to inform how well the candidate understands his subject, and what promise his research projects have long term.

    And you might scoff, impossible! Can't be done! But it has worked very well in successful businesses forever (including mine). Promotion and the assignment of managers and technicians to projects is usually based on years of working with them and seeing just how well they understand their area, and how well they perform, not by some simpleminded numerical measures in a simple formula. And often, to the extent such simple numbers are used (primarily, and thoughtlessly) it leads to bad decisions, like blaming bad numbers on an individual who has performed well but depends on others and outside factors. The numbers must go with intelligent analysis, not just simple-formula use (or the use of that formula must be based on intelligent analysis and evidence, including indirect, logic-chain evidence).

    And this also gets at the long term versus short term. As science and the world has gotten far more advanced and complex, good valuable work, and the understanding required, takes far longer. If you force people to publish every year or two (with severe implicit or explicit incentives) then they will rarely risk doing truly valuable study and work that may take a decade or more to yield a pub.

    Finally, it doesn't seem at all that there's a shortage of valuable research work that can be done. There's so much that I see that could be done, and that I have seen for 10 years, with no one doing it.

    Why not me? See the above. No one paid me to spend the very long term time to learn the skills and do the projects.

    ReplyDelete
    Replies
    1. Anonymous9:04 AM

      Just a quick and dirty comment: I think you're right. From watching years of tenure decisions, what annoys me horrifyingly is the reliance on metrics, rather than getting to know the junior person, and more importantly--looking over EVERYTHING in that person's file. Reading the publications, and the drafts of work in progress; reading syllabi, samples of student papers with comments, etc.; and thinking, hard and deeply, about all of it. Threats of lawsuits, it seems, are part of what drives the use of metrics--easier to defend in a court of law--but then the tail has throttled the dog.

      Delete
    2. Anonymous3:15 PM

      my postdoc advisor, who i consider a truly gifted molecular biologists (one nobel prize winning student, and two close to nobel level), a member of the national academy, told me that when he was chair, there was a two stage process for tenure
      the first stage had people in the field, so my advisor could shepard thru people with a small number of important papers
      however, the second stage was a university wide commmittee, wiht people in english, history, chemistry, etc who obbviously know nothing about molecular biology
      so my advisor couldn't sya, this person had only 2 papers, but they were really significant; the committe just "weighed the stack"

      Delete
  21. On the left, the seeds of distrust in science were sown in the 1960s. Science was a tool of the big corporations, the ones who insisted that lead in gasoline was harmless, PCBs were safe and DDT was completely harmless. It didn't help that Secretary of Defense Macnamara was applying scientific principles to win the war in Vietnam which is why there was all that carpet bombing and daily body counts on the nightly news.

    The left both damaged the authority of science, but also rebuilt a lot of it. DDT was banned and eagles made a comeback. Lead was banned and cars kept running. PCBs were banned and industry survived. Our air and water were safer. The take back science moved may have destroyed central scientific authority, but it had some positive results. The whole personal computer movement grew out of a bunch of hippies trying to bring computing "to the people". The Whole Earth Catalog captured this with its don't trust corporate anything, build your own and live off the grid.

    The right has always been against science. The loathing of "pointy headed intellectuals" dates back to before the founding of the republic. There were always religious fundamentalists who insisted that pi equals three and the like. It got worse in the late 19th century when modern industry absolutely required university level research and deprecated hands on know-how.

    Worse, the same people who figured out how to roll steel and temper glass were now trying to improve society. The right would not have minded a better society, but a lot of their ideas involved giving food, shelter and health care to people who didn't deserve it. If fighting the plague meant freeing the poor from fleas, the right would much prefer dealing with the plague, so they attacked science using the same approach as the left, but so far have not shown any signs of rebuilding it.

    The left challenged science and its authority of science never completely recovered. There was also the whole "there is no truth" thing on the left in which any source of information was as good as another. The right took advantage of this just as they are now insisting on safe spaces for bullies and bigots. When there is no truth, anyone can create their own truth.

    ReplyDelete
  22. Anonymous3:12 PM

    your premise is wrong
    you don't get tenure at the top schools solely for what you have done; you must also show promise for the future

    but in general , your idea that there are way more professors needing papers then there are good ideas is so old.....you have, in typical econ manner, dressed it up in a complex way.

    The real question here is, why do we keep promoting people who don't do good research ?

    ReplyDelete
  23. You are right. Trump is just a symptom.

    I used to believe Internet is the best thing ever happened to democracy. Then Internet killed democracy.

    I don't know what to believe anymore.

    ReplyDelete
    Replies
    1. Democracy is overrated. Having a cabal of 'insiders' or 'elites' pulling in levers in DC is a better alternative to having actual democracy where every hick or SJW's opinion is equally valid as mine, or Noah's.

      Online, every bad opinion gets an audience. Even supposedly open-minded reddit is increasingly a cult of personality hive mind.

      Of the 7 billion people in the world, probably 98% are still living hand-to-mouth.

      Delete
  24. There are two common enemies here. One is confirmation bias, and the other is the tendency to “generate fame” by creating counter-intuitive results.

    I recently saw some scientific research described in the “mainstream media” regarding gluten. The research produced a result that was contrary to my long-held belief, so I eagerly sought out the original study, to determine if my old assumption was incorrect. Before I even finished the quick summary, I had spotted three methodological flaws that would normally invalidate such research (the biggest being a self-selected sample population).

    So now I have to decide if I am being anti-science. My original belief was based on a sample size of me, which was also pretty much self-selected. Had the new research confirmed my opinion, I would have (most likely) accepted it uncritically and even defended the sampling flaws.

    I eventually decided that the study authors knew (in advance) that this study would not settle the question. The fact that they got publicity for drawing a broad conclusion from a narrow study is just icing on the cake. They can now get funding to repeat the study, or hopefully a better one, for the remainder of their careers.

    ReplyDelete
  25. And, Noah, apparently you think there is no BS climate science to be criticized? Climate scientists don't suffer from bias or financial conflicts of interest, only economists, psychologists, and rest of us mortals?

    ReplyDelete
  26. What role do the incentives for JOURNALS play. Journals currently have very weak incentives to publish replications (let alone negative results!). Therefore, assistant professors have very weak incentives to work on replications. Therefore we get a lot of interesting results, but results which might be derived from misspecified models, dubious assumptions, or poor statistical methods.
    Correspondingly, researchers at large institutions have weak incentives to share datasets to make replication more feasible. A good dataset is a cash cow that can generate dozens of publications.
    Incent journals to publish more replications, share more data, and the opportunities for publication will increase geometrically.

    ReplyDelete