tag:blogger.com,1999:blog-17232051.post501824115534162824..comments2024-03-18T22:32:52.802-04:00Comments on Noahpinion: A bit of pushback against the empirical tideNoah Smithhttp://www.blogger.com/profile/09093917601641588575noreply@blogger.comBlogger24125tag:blogger.com,1999:blog-17232051.post-92104009452573285752015-09-30T07:04:07.321-04:002015-09-30T07:04:07.321-04:00Wanky is a reasonable synonym for pompous. Glad to...Wanky is a reasonable synonym for pompous. Glad to see Noah picking up essential parts of British culture.weareastrangemonkeyhttps://www.blogger.com/profile/07725580843974605480noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-25547361801541532172015-09-29T15:07:33.680-04:002015-09-29T15:07:33.680-04:00" In a later, more pompous/wanky post..."..." In a later, more pompous/wanky post..."<br /><br />You must mean "wonky"...Cryptic Crosswordshttps://www.blogger.com/profile/01738199550144981751noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-30923681953387492262015-09-29T07:40:39.311-04:002015-09-29T07:40:39.311-04:00OK, so empirical data of the kinds economists are ...OK, so empirical data of the kinds economists are generally able to collect can give one only limited information about the likely outcomes of the relevant alternative policy choices. But a priori theoretical reasoning unconstrained by empirical data gives one even less information about the likely outcomes of the relevant alternative policy choices. So I guess the conclusion is that the methods available to economics do not provide economists with a lot of information about the likely outcomes of the relevant alternative policy choices, and that economists therefore do not possess a lot of information about the likely outcomes of the relevant alternative policy choices.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-17232051.post-76957220018025752452015-09-28T21:10:51.408-04:002015-09-28T21:10:51.408-04:00Well, sure. Other techniques have problems, but yo...Well, sure. Other techniques have problems, but you brought up IV specifically...Ibnyaminhttps://www.blogger.com/profile/02200468798337178145noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-79394616666609058492015-09-28T20:49:46.269-04:002015-09-28T20:49:46.269-04:00To me, it seems that economists expect too much fr...To me, it seems that economists expect too much from empirical papers. Reinhart-Rogoff is a classic example. The profession literally took *one* paper, and treated it as the truth to the point where economists were torturing their data to get a similar result. Hard sciences are also limited by the conditions that they set on any given experiment. In a way, instruments as economists use them and control variables as scientists use them are analogous, although they obviously have a far greater control than we do. The problem is that we often fail to take the results of a body of literature to do meta-analytical work to see what conditions impact various theories, and recraft our theories to fit that. Ultimately, any science is only as good as the questions scientists choose to ask, and at present, most economists are trained not to ask certain questions or to regard results that don't agree with the theory as spurious.Vulgar Economicshttps://www.blogger.com/profile/02732213110359339737noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-78586574866356733532015-09-28T19:01:18.132-04:002015-09-28T19:01:18.132-04:00These seem to be complaints about over-interpretat...These seem to be complaints about over-interpretations of and extrapolations from limited empirical data, not really pushbacks against the empirical tide. The only cure for empirics is more empirics.Tom Warnerhttps://www.blogger.com/profile/11247836188106712069noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-6405544829163086932015-09-28T18:18:32.806-04:002015-09-28T18:18:32.806-04:00I think you're right - there's definitely ...I think you're right - there's definitely the potential for data mining here. But instruments are so hard to think of that it seems like less of a problem than for many other techniques out there.Noah Smithhttps://www.blogger.com/profile/09093917601641588575noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-62442926513072840222015-09-28T18:16:44.715-04:002015-09-28T18:16:44.715-04:00It put Romer off, not me! I just thought his ideas...It put Romer off, not me! I just thought his ideas were bad.Noah Smithhttps://www.blogger.com/profile/09093917601641588575noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-76694508255945807982015-09-28T16:46:07.675-04:002015-09-28T16:46:07.675-04:00I think that most of the time the instruments test...I think that most of the time the instruments test as weak or as correlated with the error. They throw them out and try again. The trying again is a problem. The researcher has already seen the data and this will influence the choice. How many instruments are rejected subconsciously without running any tests? This is a problem. And how would anyone critiquing the results know if they had run through thousands of instruments? Ibnyaminhttps://www.blogger.com/profile/02200468798337178145noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-42839379383650009892015-09-28T16:28:57.594-04:002015-09-28T16:28:57.594-04:00Is your problem with theory, theory itself or theo...Is your problem with theory, theory itself or theory that has not been properly tested? <br />I see two types of theory. Type 1 are models that make qualitative but not quantitative predictions. I see those models as pretty limited. One exception is that qualitative theory can sometimes help with identification. (For example, sign restrictions or Minnesota priors in VARs.) Clearly theory helps can also evaluate an empirical exercise. If the data says demand slopes up we should be skeptical of the results.<br />The second type of theory are those that make quantitative predictions. These are the most useful and interesting, but are also the most abused. I see the problem is that we frequently write down models and then assume they are true to make empirical predictions or to answer policy questions. That is a problem with our methodology not with theory. We need to have a metric to evaluate a model before treating its predictions seriously. <br />At the end economics is mostly about identification. Natural experiments are hard to come by and valid instruments are tough to find. Properly used theory is an adjunct to data and needs to be integrated into data. Anonymoushttps://www.blogger.com/profile/06522136998161060309noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-63002248871884183792015-09-28T16:28:40.295-04:002015-09-28T16:28:40.295-04:00Noah you are right that is part of effective rheto...Noah you are right that is part of effective rhetoric. Ed Prescott's black and white crusading style put you off. But to quote a wise man "maybe that's just how you have to be if you want to get things done in the sciences!" Anonymoushttps://www.blogger.com/profile/06522136998161060309noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-81920342995985811832015-09-28T16:07:03.333-04:002015-09-28T16:07:03.333-04:00Ben, you think people really run through a whole b...Ben, you think people really run through a whole bunch of candidate instruments? I think they're hard enough to find or think of that that probably only rarely happens (except in time series, when you're using lags as instruments...ugh).Noah Smithhttps://www.blogger.com/profile/09093917601641588575noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-81679273058878108762015-09-28T14:56:24.062-04:002015-09-28T14:56:24.062-04:00You don't think saying "I have not seen&q...You don't think saying "I have not seen" is a qualification?<br /><br />Also, I don't think what you just said is a good example of thinking about scope conditions. Sure, we know the results are local. How local are they? How small is a small change? I haven't seen much effort put into answering those questions, and your comment fits with my own observations.Noah Smithhttps://www.blogger.com/profile/09093917601641588575noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-68568790299491569172015-09-28T14:49:44.869-04:002015-09-28T14:49:44.869-04:00"I have not seen economists spend much time t..."I have not seen economists spend much time thinking about domains of applicability (what physicists usually call "scope conditions"). But it's an important topic to think about."<br /><br />Noah, this is completely ridiculous. Ironically with respect to your second item, IV is where this has been done the best--Angrist and Imbens showed that IV identifies a LATE (*local* average treatment effect) for compliers and not an ATE for the population, which directly codifies these "scope conditions." I'm sure there are people on the internet who don't realize you're completely out of your domain--is it too much to ask for you to even slightly qualify ridiculous sweeping statements like in most of this entire post?Pseudonymousnoreply@blogger.comtag:blogger.com,1999:blog-17232051.post-23036381506519987662015-09-28T14:42:53.579-04:002015-09-28T14:42:53.579-04:00A particularly egregious recent example from healt...A particularly egregious recent example from health research claimed that "high" exercise was worse for health than "moderate" exercise - after conditioning on BMI, blood pressure, blood lipids, and a bunch of other stuff that most people would assume is instrumenta.Phil Koopnoreply@blogger.comtag:blogger.com,1999:blog-17232051.post-79946041954291464782015-09-28T14:38:39.055-04:002015-09-28T14:38:39.055-04:00You could say exactly the same thing about confoun...You could say exactly the same thing about confounding variables, but practice of blindly conditioning on possible confounders remains very popular. Every decision to categorize a variable as confounding, instrumental, or irrelevant equates to assuming a causal model; there is nothing special about IVs in this respect. One has the choice of explicitly identifying one's causal assumptions or ignoring the issue and hoping nobody notices.<br /><br />It is possible to find combinations of datasets and causal models that produce "Simposon's Paradox machines"; by progressively conditioning on factor 1, factors 1 and 2, factors 1, 2, and 3 one can reverse the conclusion at each step. It is common for researchers to say "our findings are robust to conditioning on factors A, B, and C", and critical attention usually focuses on unnamed factors D, E etc. But how many papers explicitly state that their findings hold up when conditioning on {A}, {B}, {C}, {A,B}, {A,C}, {B,C}, and {A,B,C}?Phil Koopnoreply@blogger.comtag:blogger.com,1999:blog-17232051.post-75692631179147809762015-09-28T14:34:28.117-04:002015-09-28T14:34:28.117-04:00By random chance some instruments will test falsel...By random chance some instruments will test falsely as strong instruments and not be correlated with the errors. If you run through enough instruments, you will necessarily identify false positives. You may not even realize this is what you did. Ibnyaminhttps://www.blogger.com/profile/02200468798337178145noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-14475773153824946612015-09-28T14:30:24.860-04:002015-09-28T14:30:24.860-04:00A&P's crusading rhetoric seems to have put...A&P's crusading rhetoric seems to have put off some folks, but maybe that's just how you have to be if you want to get things done in the sciences!Noah Smithhttps://www.blogger.com/profile/09093917601641588575noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-63645344408536751552015-09-28T14:29:51.109-04:002015-09-28T14:29:51.109-04:00I disagree. Sure, there are other instruments you ...I disagree. Sure, there are other instruments you could have selected. But so what?Noah Smithhttps://www.blogger.com/profile/09093917601641588575noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-2853684053829652015-09-28T14:29:31.178-04:002015-09-28T14:29:31.178-04:00Yeah.Yeah.Noah Smithhttps://www.blogger.com/profile/09093917601641588575noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-7609799037768489622015-09-28T14:06:28.065-04:002015-09-28T14:06:28.065-04:00I agree Amy. Structural approaches do not avoid th...I agree Amy. Structural approaches do not avoid the locality limitation problem. Ryan Deckerhttps://www.blogger.com/profile/03060713323484397524noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-69972950559104915852015-09-28T13:58:51.501-04:002015-09-28T13:58:51.501-04:00The locality argument is a major issue with all of...The locality argument is a major issue with all of econ, in my opinion. I came over from engineering and spent the first year or so of grad school just being like, are you guys crazy? As someone who writes code, the first thing I do when I see a formula is plug in zero and see if it breaks. In econ, it always breaks. Are capital and labor substitutes? Not at the limit. <br /><br />But I've come to accept the "at the margin" argument for everything. :) My chief complaint at this point is that economists don't make that clear when talking to the public. For example, should we have fiscal stimulus? Well, at the limit can we just do more and more fiscal stimulus until GDP grows at 10% per year? Probably not... so just explain that. Otherwise people reject the whole model out of hand because it doesn't pass those kind of common-sense tests.Amyhttps://www.blogger.com/profile/11573392015410961658noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-11259386815673246242015-09-28T13:33:25.955-04:002015-09-28T13:33:25.955-04:00While I admit that my blogging style is exaggerate...While I admit that my blogging style is exaggerated, I think that the people who are over the top about IV are Angrist & Pischke!Angushttps://www.blogger.com/profile/03656436431053306500noreply@blogger.comtag:blogger.com,1999:blog-17232051.post-68159338693814545032015-09-28T13:30:31.640-04:002015-09-28T13:30:31.640-04:00I don't think Kevin is being hyperbolic and I ...I don't think Kevin is being hyperbolic and I think he is right to reject IV. There are arbitrarily many instruments you could have selected. Your mind rejects many of them subconsciously. Why? You don't know. The instruments you do select may be the result of your own brain fooling itself into selecting instruments that pass the tests given the data. And how do other researchers know if you ran the equivalent of batch file regressions on different sets of instruments until you found ones that rejected the nulls? The technique lends itself to intellectual dishonesty. Ibnyaminhttps://www.blogger.com/profile/02200468798337178145noreply@blogger.com