Sunday, August 11, 2013

A healthy side effect of High Frequency Trading?

High Frequency Trading is nearly universally reviled. The Tournament Externality is just too obvious; it yields no social value to bring a piece of information to the market 1 millisecond before it would otherwise have arrived, but people are apparently spending lots of money in an effort to do so. It can't be efficient to commit our best and brightest minds to beating each other to the punch by 1 millisecond. Because of this, many have suggested a small Tobin Tax to curb HFT. Others have suggested batch auctions.

As for me, I've never been that worried about HFT, because I viewed the costs as small. HFT profits have been falling steadily in the few years since the technique became popular, from almost $5B in 2009 to just over $1B in 2012. Trading volume is down too. And I'm not very scared of flash crashes. So I viewed HFT as a very minor nuisance at worst.

But what about the benefits of HFT? The one usually cited - "providing liquidity" - seems a minor benefit. Minor benefit, minor nuisance. Then there's this interesting paper by Douglas Cumming, Feng Zhan, and Michael Aitken, which suggests that HFT actually drives out market manipulators. Robots defeating sneaky underhanded Wall Street fixers? That sounds good to me! Though of course HFTs can engage in manipulation of their own.

But when I read this article, I thought of another possible benefit of HFTs. The article, by Eddy Elfenbein (excellent name, btw), is about day traders being made obsolete by HFT robots:
I’ve got bad news for all you day traders out there. Just like assembly-line workers, switchboard operators, copy clerks, and hand weavers before you, you’ve now been automated out of existence... 
What [HFT] means for you is that now more than ever, day trading is a fool’s errand. If you were ever tempted to enter the fray, recognize once and for all that those banks of computers chugging away at their algorithms have you hopelessly outclassed. There’s simply no way an individual human being can compete against a fleet of CPUs that rivals NASA’s. Do not, I repeat, do not try it. You will lose.
This gave me a new, positive vision of HFTs. If day traders across America really do believe what Elfenbein is saying, and if they really do stop day-trading and start doing long-term fundamental investing, then the HFT robots are doing the Lord's work.

Why do I say this? Because here's the thing about day traders: Even before HFT, day traders consistently lost money. Actually there is a small number of day traders - maybe 1% - who make money day trading. But the rest just lose, and lose, and lose. The evidence is not ambiguous.

As for the reason that human day traders lose money, there are probably several factors. Humans are overconfident. We don't benchmark our performance very well. Mental accounting makes us ignore trading costs (which tend to gobble up whatever profits a day trader thinks he's making, and more). Self-attribution bias makes us think that our gains came from skill and our losses came from luck. The basic fact is: You, a human, make a lousy, lousy HFT algorithm. And if robot HFTs make you realize this fact and quit trying to be a human HFT, then they are doing good for both you and for the world.

Thinking of this, I'm starting to realize why Zero Hedge hates, hates, hates HFTs. With its shirtless Brad Pitt avatar and its tough-guy language, Zero Hedge caters to testosterone-junkies. Testosterone junkies are young overconfident dudes with a little money in their pocket, who maybe work day jobs on Wall Street, or maybe just got their law degrees, or maybe work as engineers and think they'll be the next Steve Jobs, or whatever. Being young, overconfident dudes, they think their intelligence and energy and sheer will-to-power will allow them to take on the world. And when it comes to their finances, taking on the world means beating the market by seat-of-the-pants day trading.

We've all met these guys.

So of course these guys, and the websites that cater to them, despise HFT. Not because it crashes markets, not because it wastes money on pointless "information tournaments", but because it takes away their dreams of outwitting the world and landing that big score.

To which I say: Sorry, dudes. Much as it pains me to say this to anyone, your day-trading dream was always kind of an illusion. You were always just going to eat trading costs and die. Here's my advice: Don't give up on crazy world-conquering dreams, but try to switch to one that creates real value for society instead of just adding noise to financial markets and feeding the coffers of quant funds. Give up the day-trading dream, and focus on being the next Steve Jobs.


  1. Yet, I believe the true cost of HFT is incurred by institutional traders. Say that an institutions need to liquidate their holdings, they will find a way to optimally execute their trades in order to minimize execution cost but we know that HFTraders are somewhat efficient at detecting order flows and do "opportunistic front-running". That, to me, is the real social cost of HFT.

  2. See for example this paper by Adam-Clark Joseph on exploratory trading: we can get the real sense on how HFT front-run

    1. Anonymous11:55 AM

      How is that front running? In what sense are the people, who placed the resting (passively or actively) orders, clients of the people who are running automated algorithms for trading.

      (From Wiki:
      Front running is the illegal practice of a stockbroker executing orders on a security for its own account while taking advantage of advance knowledge of pending orders from its customers.

      Back in the nineties, when specialists filled orders, there were plenty of examples of front running and other double dealings. It was a opaque market, more akin to a snake pit. With computerized trading, the field is somewhat more even. You *can* write your algorithm, and connecting them to live trading is far far cheaper than before. Going to HFT market making (whether official or not) pinched market making profits in large sell side institutions. For evidence look at investment bank balance sheets and look at how often Equity S&T profits beat FICC S&T profits now vs before.

  3. Anonymous4:16 AM

    It's time to rename your blog, NoahIdea.

    Its no surprise your an academic and have no true connection to the financial world. As a successful and profitable intra-day trader I find your explanation for why "human daytraders loose money" laughable.

    The world is full of successful intra day traders; traders at banks, hedge funds, proprietary trading firms & individuals working from home. These people work hard to develop and continually refine the skills necessary to be one step ahead of the rest. You and the jokers you reference claim its a "suckers game"

    Its the failed traders that write blogs like this, stroking their egos and justifying their failures. If you think sharing an office with other growth minded individuals, learning & working as a team + being rewarded for your performance with 7 figures a year is a suckers game. You truly have no idea!

    1. This comment proves that the apostrophe is redundant. Perfectly clear without a single one. I'm not bothered whether it's deliberate or not btw.

    2. Was this reply a deliberate attempt to satisfy Noah's stereotype or should I be in awe at the irony?

    3. Was this reply a deliberate attempt to satisfy

      If you have to ask, you would not understand the answer.


    4. Anonymous8:25 PM

      The fact so many people are able to make enourmous, market-beating profits with unending consistency casts doubt on the EMH.

  4. In the end this argument amounts to a year old article by Jerry Adler that amounts to a high tech version of Milton Friedman's old argument as to why forex markets would be stable because speculators would necessarily lose money, a result disproven by DeLong, Shleifer, Summers, and Waldmann over 20 years ago formally, and long before that in real life. Adler's argument is mere assertion: "Man, these algos are getting really damned fast!" along with a snide remark that the guys at some quant conference only own one tie, if that many.

    Well, heck, I am not sure Mark Zuckerberg owns a tie even today, and I remember a session on econophysics at AEA over a decade ago when non-tie wearing Doyne Farmer, who sold his Prediction Company to UBS for a large wad of money sneered at economists because the latter wear ties because they are so stupid that they must impress bankers, although Benoit Mandelbrot was sitting in the room wearing a rumpled suit with a tie. Of course, the more serious point is that if a quant in the room makes enough, he'll leave the damned room, so a truly stupid argument.

    The real warning is the ongoing existence of the flash crashes. There is nothing whatsoever out there or in Adler's piece that proves that the greater accumulation of ever faster HFTs proves that they are ultimately stabilizing on any sort of fundamental in the way the Uncle Miltie thought the fundamentalist traders would be in forex markets. There is no guarantee or proof of this whatsoever. Instead, we have that the possibility of really awful crashes coming out of nowhere may be increasing (Rajiv Sethi has also written on this at length in his blog).

    There are three time horizons in trading. In the long run (over a year), economic theory is correct: buy and hold index funds because random walk wins. In the intermediate run, behavioral strategies win, although one should get an uberinformation cruncher to roll in and out appropriately the anomalies as they appear and disappear in the algorithms. Then there is true HFT, which is the realm of HFT and ultimately based on trying to beat bid-ask spreads.

    One of the best of those traders is Jean-Philippe Bouchaud, a physicist with a successful fund in Paris. He is clear that fundamentals do not matter to him. He does not care. In that regard he is like Post Keynesian Paul Davidson, who says we can never know true fundamentals because of Keynesian uncertainty. All Bouchaud cares about is nano-second to tera-second patterns, and there is nothing at all that says that the winning strategies for this sort of strategy are necessarily stabilizing.

    So, it may be getting harder for the day traders to beat the algo quants, but it is not impossible, and nothing in either of these articles shows this at all.

    1. Nathanael12:04 AM

      "In the long run (over a year), economic theory is correct: buy and hold index funds because random walk wins."

      Not quite right, as my father Anil (who you know) could tell you.

      There's actually a longer time horizon: over the 5-year-plus horizon, it's quite easy to beat random walk with fundamentals research and stock picking. Economic theory is simply wrong.

      Incidentally this has been Warren Buffett's principle.

      Fundamentals research requires actual knowledge of the specific details of specific business operations within specific industrial sectors, of course, which is why so few people do it -- it's actual work. It even requires analyzing the behavior of individual CEOs and Boards of Directors to figure out whether they are going to manage their businesses competently or stupidly.

      It's an example of beating the markets through having information which the other people in the markets don't have, not because the information isn't public, but merely because very few bother to research it.

      Of course, if you don't have the time to do the research, index funds are the best you can do. If you happen to be intensively researching particular sectors for other reasons anyway, however, you can beat the indexes over the long run just by avoiding the mismanaged companies which are on a glide-path to bankruptcy.

      (Incidentally, there was a study a few years ago which debunked the idea that holding a stock index was consistently better than holding cash. Turns out, in the 19th century, it wasn't -- the bankruptcies made the difference. The bankruptcies had been omitted in the previous longitudinal studies which gave the false result. Of course, *as usual*, I don't have the reference on me. In short, in order to reliably do better than cash in a non-inflationary environment, you have to figure out how to avoid investing in companies which will go bankrupt.)

      I appreciate the rest of your comment, which I largely agree with.

  5. Meant to say "true HFT, which is the realm of econophysics." Also, we should speak of winning "tactics for this sort of strategy."

  6. Noah, you can't evaluate social costs of HFT based on profits alone, you have to take into account expenditures including compensation (Malyshev alone got 75 million from Citadel in 2008), infrastructure costs (such as high speed fiber-optic cables between major global trading centers), energy (cooling is a major variable cost), equipment (replaced routinely every few months), proprietary data costs (Nasdaq's TotalView ITCH), colocation services, etc... profits are just the tip of the iceberg. Gross revenues are key.

    1. Of course, but shrinking profits should hopefully reduce investment in the strategy, economy-wide.

    2. Not necessarily. Profits can shrink through entry of new competitors, and hence greater duplication. Diminishing profits will slow entry but that doesn't mean that low profits correspond to lower gross revenues.

      For example, look at output (and hence total production costs) versus profits in a basic Cournot model as you change the number of firms. If there's an entry cost, firms will enter till profits fall to the level of entry cost. Profits decline with each entry while aggregate production costs and output rise. The difference is that in the Cournot model there is some consumers' surplus to compensate so welfare rises. Here there is none.

      Furthermore, greater entry can result in more price volatility. Forget the flash crash, what matters is routine volatility, especially in small and mid caps, that can affect portfolio choices. Market generated risk raises aggregate risk and shifts portfolio weights to less risky alternatives. The result is less risk-taking, less growth.

    3. But, Rajiv, the flash crashes are the clearest evidence that HFT does not necessarily reduce volatility, however measured.

    4. I agree Barkley, I was just trying to say that even if Noah doesn't care about the flash crash itself, he should be concerned about volatility more generally. To me the flash crash is highly informative - I called it an extreme version of a routine event in a previous post - but if he considers it pathological he should look at other measures of volatility and correlation.

      My main point, though, was about resource waste - to say that lower profits are indicative of lower waste is just not correct.

    5. Rajiv, I agree with you completely. However, I am not sure if HFT increases volatility at daily or monthly frequencies, which is what I think matters for corporate finance (investment decisions are not made in milliseconds).

      I think that you're right about the resources wasted on HFT. However, I think the waste of resources is due to a pernicious culture among (especially) pension funds, and also insurance companies, that overvalues active trading. Pension funds are wasting people's money by giving them to active traders, so that's where we need to focus our reform efforts.

    6. Well, since we are getting into this, Noah and Rajiv, it is indeed true that there is no unambiguous measure of volatility, which can vary across time horizons. Increased volatility at one time horizon might be associated with decreased volatility at another, quite aside from the problem that simple variance, the standard measure, may not fully capture volatility, which clearly must account for fat tails and skewness and all that stuff.

      Indeed, for the last several years there has been a major meme in these discussions that shorter term or more local reductions in volatility might be associated with longer term increases in fragility, or global volatility/instability. The lit that things like HFT ought to increase efficiency and rapidity of convergence to (at least short-term) equilibria is simply enormous, this lit extending to adding more and more risk spreading assets as well. Needless to say, the Minsky Moment following on the collapses of Lehman and AIG and the related pyramids of higher order derivatives has sharply reminded us that supposed short-run stability and efficiency may be at odds with longer term resilience. Indeed, that these might be at odds in general is a principle from ecology enunciated by C.S. Holling in the early 70s.

      Oh, and, of course, Rajiv's argument about inefficent use of resources for these sorts of HFT wars is correct.

    7. But here's my point about resource wasting.

      Suppose HFT profits dry up completely.

      Some pension funds will still fling money at HFTs, to be absorbed by the HFT managers and by the funds' costs.

      But mostly, pension funds will start flinging money at whatever other hot, trendy, expensive "special" classes of actively managed funds have been enjoying good returns over the last couple of years.

      What we need to do is stop the pension funds from flinging the money at third-party managers, and force pension funds to either be passive investors, or sources of patient, activist capital.

      THAT will dramatically cut down the waste.

    8. Can we just do away with pension funds? Why should I be encouraged (by the government through tax advantages) to invest in funds which routinely act in the best interest of money managers and not in my best interest.

      Without the government tax advantages, I would be better off investing in Indexes.

    9. Anonymous12:03 PM

      Similarly, you should also count the improvement in computer architecture among the social benefits of HFT trading (just like porn was instrumental in developing good network infrastructure, the twin forces of advertising and HFT are instrumental in paying for improvements in computer architecture, specifically memory, disk, IO). I am sure if DWave like companies actually succeed in proving that they are doing quantum computation effectively, HFT will pay for their further development.

  7. A question I've been puzzling over: why do we even need intra-day trading? Let's just agree to trade once a day. Isn't that enough for any purpose other than gambling?

    This kills a lot of birds with one stone, putting day traders, HFTers, etc. completely out of business. And it doesn't penalize anyone else, unlike proposals to tax transactions.

    1. HFTers, Day Traders, etc, are just looking for chances at arbitrage. In an "ideal" market, aka a market at full efficiency, no such opportunities can exist. Full efficiency clearly isn't possible.

      Thing is, like all man-made systems, financial systems are follow-the-leader systems of social status--they have the potential to become self-reinforcing. So HFTers seeking risk free returns from market noise now number among the main producers and amplifiers of market noise. I imagine there are algorithmic traders right now working on ways to detect, predict, and exploit arbitrage opportunities stemming from HFT noise.

      When I was living in the world of dynamic control theory I saw a presentation by a chemical-mechanical engineer who developed and proved a mathematical method for extracting information from noise. If I'd only had 3-5 more years of funding and time I would have loved to explore and master his work. Ah, but for the vagaries of capitalism. Sigh!

      Just remember that personal wealth + income have little to do with actual skill or intelligence and much more to do with social status and you'll be fine. Professor Smith's day-trading testosterone junkies know this intuitively even if they play it out via Cheetoos and a computer screen. In their minds they conquer the competition and escape to a mythical realm filled with shiny cars, horny models, strong liquor, and early death. Sounds good to me. Some of them make it--, for example. 99.999% have to settle for some lost cash and Orange Penis Syndrome. Good laws and market discipline can fix many things, but they can't fix stupid.

    2. Anonymous7:37 PM

      Once a day? You'd move the world back to the dark ages.

      After the earthquake in Japan, you would expect me to wait 24 hours before hedging risk? Deals contingent on a commodity price could only happen once the new price was established every 24 hours.

      Do you work in Washington?

    3. Anonymous12:05 PM

      Once a day? What are you, some kind of evil trader?

      Once a year is good enough. We can do twice a year on leap years.

  8. Anonymous12:48 PM

    I have often wondered why there may be a rationale for high frequency trading on stocks. How about this rule instead? The buyer of the stock has to hold the stock for a mandatory period of one day/24 hrs after which he can do whatever he wants with it. Somehow, this seems to be a simple way to mitigate (if not, eliminate) pure speculative attacks and market panics... I cannot seem to pin this down exactly. But low frequency trading on stocks might be welfare improving in the long run.

  9. High Frequency Trading seems to exploit problems in the structure of the markets. We do not need to outlaw HFT, as Noah seems to suggest. We should however look at the structural problems in the market that HFT exploits and eliminate them. Things like reporting trade information to one investor before reporting the information generally to the market and having rules allowing complex trade structures which allow HFT firms to jump the order queue.

  10. I'm a supporter of the interval batch trading approach, the article linked above neglects to note that batch intervals need to be randomized else the HFT guys will work on timing the ticker (presuming the other bids in the batch are visible).

    By randomizing the intervals you destroy the high frequency information, rather than just attenuating it a bit. We can argue about how much information should optimally be destroyed, I would be making the mean interval time approx 1 minute.

    Also, I'm not a supporter of government interference in this, it's a matter for the exchanges to deal with. Fundamentally, exchanges want to attract, fresh investment money and they won't get that while the punters are expecting to suffer higher costs caused by HFT. Sooner or later one exchange will do it, and free market competition will sort it out. Government involvement just messing things up worse.

  11. Who has done mor overall harm in the last 10 years? Day traders or economists. Maybe your priorities are wrong

    1. Anonymous8:21 PM

      daytraders blame economists for losing money

  12. Phil Koop6:31 PM

    Good heavens! First the libertarians, now the day traders. In one weekend! You must be developing an extraordinary tolerance for derp.

    1. You must be developing an extraordinary tolerance for derp.

      More like he has lost all patience with it.

  13. Anonymous7:45 PM

    Daytrading & HFT are two completely different things. Weaving them together in this article is deceptive.

    If I buy on the open and sell on the close i'm "daytrading" and that's clearly not high frequency.

    I work at a Hedge Fund that takes on risk within the day and overnight. Our intra-day trades can last hours, what harm are we doing?

  14. Anonymous8:19 PM

    Agree 100% Noah. HFT doesn't pose a threat because over the longer run (months & years) stocks will still rise to their fair value and because HFT works both way; either buys or sells can be traded with high frequency, not just sells. Having lots of high frequency traders smooths out inefficiencies in the market making the process more democratic for average investors and the extra volume helps keep the bid/ask spread narrow. If you buy 1000 shares of a stock with a 50 cent spread you've already lost $500 if you try to sell. Computerized trading hasn't made stocks more susceptible to crashing. For example, stocks suffered high drawdowns in various panics in the 1800's and 1900's.

    1. Anonymous8:28 PM

      But disagree with the part about day traders not making money. too many counter exampes

    2. Nathanael11:50 PM

      This is just wrong, but read my comments near the bottom for why it's wrong. Yours is a very shallow understanding of the potential dangers to markets of front-running and instability.

  15. HFT is just the old wire con. It makes its money by front running trades and taking a sweet chunk of the spread between the bid and ask. It takes this chunk from long term buy and hold types and from day traders alike. Since day traders trade more often than buy and hold types, HFT will take more from them, but it won't put them out of business. They are trading against players at a lower time scale. HFT just imposes a tax on all traders including other high frequency traders, now that most firms have their own HFT access. (The wire con really only works well when one party is playing the wire.)

    The main downside of HFT is that we haven't reset the safety mechanisms yet. In the old days, there were specialists on the floor, and their job was to ensure stable trading. In exchange, they got to take a sweet chunk of the spread between bid and ask. The specialists are long gone, and the idea of a businessman having any responsibility beyond profit maximization is more alien than Kirk's friend, Spock. There is no reason we cannot all drive at 200 miles per hour. With the appropriate safety features, it shouldn't be any more dangerous than puttering along at 25. The problem is that we don't have the proper 200 mph safety features in place yet.

  16. Have you ever tried to cross your own trade? Say the bid/ask is 25.00/25.10. Submit a bid in one account with a limit at 25.05, then submit a sell in another account with a limit at 25.05.

    Think you will cross your own trade? Think again. Typically, the sell will get executed at 25.0501, and the buy will not get executed.*

    As some other commenters explained, HFT exploits flaw in the market structure. It intercepts order flow and executes those orders against its own book, without ever posting a publicly visible limit. In doing so, it severely punishes those who DO post public limits (through adverse selection), and drives liquidity out of the market. In turn, the lack of a deep public limit book is the reason for flash crashes.

    In the above example, my buy order at 25.05 was the victim of adverse selection. It was the best bid on the market, but it was intercepted by an hft machine offering a centi-penny of price improvement.

    Please note, I have actually conducted the above experiment on multiple occasions. It is not a hypothetical example.

    1. Anonymous2:53 PM

      Hey Steve,

      What you attempted to do is called "painting the tape" aka market manipulation and its illegal. Your brokerage probably doesn't let you do it because they don't want to get bitch-slapped by the SEC. I suggest you don't try it anymore!

    2. Hey Anonymous,

      Different brokerage accounts! You must not have read my comment, you didn't even address the significant market structure flaws that allow HFT to jump in front of the existing order book.

      And it's not manipulation -- small trades, middle of bid-ask, same price as last trade, legitimate purpose, etc. But the market structure guarantees that HFT will always collect a rent from one half of the trade.

      Please go back into your HFT shill cave.

    3. Anonymous8:22 AM

      You are correct, I missed the different account part. I'm still pretty sure that it was your broker who price-improved you not an HFT though. Unless you count your broker as an HFT, which is fine but just different than how the term is generally used. If you trade without a broker I guarantee that you can cross yourself. I even think you can cross yourself with a broker (or at least by using 2 different brokers). All you have to do is make sure that your bid is actually the best bid (i.e. it shows up in the NBBO) and then you can hit it from the other account. If you are alone at the NBBO you should cross yourself a decent % of the time though sometimes your broker might price-improve the sale. I don't recommend trying it though!

      Anyway, I could be wrong. What market-structure flaw do you think there is that allowed an HFT to trade with your bid? What strategy do you think the HFT was pursuing? I'm bored so I wouldn't mind trying to figure out if your theory is plausible.

    4. If big retail brokers like Fidelity and Vanguard are in the business of prop trading that would be news to me. It happens with every brokerage I've ever used. I assumed it is "flash trading" where the machines are given a millisecond sneak peak to decide if they want to execute the order against their own book before the order hits the exchange. Whatever it is, it disadvantages publicly displayed liquidity (through adverse selection), driving people into dark pools.

      Also, I see games played with exchange fragmentation. The prioritization and trade through rules are a mess. Suppose I submit a buy on ECN A. I am the national best bid. A machine sees that and submits the same order on ECN B. Ok, fine the machine is second in line. Then the machine submits a tiny bid on ECN A in order to update the time stamp, and bumps me to bottom of the queue. Voila! The machine on ECN B is now first in line for execution at the nbb, even though its order was a later copy of mine.

    5. Anonymous11:10 AM

      That is internalization. Your order never hit the market and wasn't touched by a HFT firm, unless your broker sold the flow to them.

      Blame your broker not HFT.

    6. Anonymous11:36 AM

      >If big retail brokers like Fidelity and Vanguard are in the business of prop trading that would be news to me

      Well, you learn something new every day. I only bothered to look it up for fidelity. This website:
      claims that they do price improvement on 85.1% of customer orders.

      flash trading is basically dead. Everyone but direct edge stopped offering it a few years ago when it became controversial. But a flash order was always a voluntary order type. No one got to see your orders ahead of time unless you decided to let them.

      >Suppose I submit a buy on ECN A. I am the national best bid. A machine sees that and submits the same order on ECN B. Ok, fine the machine is second in line. Then the machine submits a tiny bid on ECN A in order to update the time stamp, and bumps me to bottom of the queue. Voila! The machine on ECN B is now first in line for execution at the nbb, even though its order was a later copy of mine.

      It doesn't work that way. Its actually worse than you think! There is no time priority between ECNs just price priority, and that only for displayed orders. If I copy your order on another ECN I can get filled ahead of you. It doesn't require any funny games with timestamps.

      In general, I agree with you that the micro-structure of the US equity market is horribly complex, confusing and annoying. I don't think it hurts retail investors but it can make life difficult for institutional investors and there is a whole industry of "smart routing" technology or services that wouldn't need to exist if there were fewer liquidity pools. I think that the existing structure benefits larger HFTs by raising the barriers to entry from a technology/knowledge perspective. Its nowhere near the size of the advantage that the specialists used to have though.

    7. Interesting, I guess I do learn something every day. So when if the national best bid is 25.05 and Fidelity executes my 100 share sell at 25.0501 (for one penny of price improvement of the entire order), who is taking the other side? Someone else at Fidelity? And why is the price improvement always only 0.0001?

      My general observation is that a publicly displayed limit order is rarely executed except under adverse selection, i.e., a gap down post an executed buy order, or a gap up post an executed sell order. There is usually lots of volume at my price prior to my execution, though, even though I was the first and best publicly displayed limit. I never leave standing limits for that reason.

      Usually I just pay the 0.0099 toll to the toll keeper (whoever that is) for an order that can be immediately executed.

      Perhaps you are right that it is less costly than the alternative, but I generally assume I am paying a 0.0099 "commission" to people who are better able to arbitrage the market structure.

    8. To make my point more succinctly, suppose the nbbo is 25.05/25.06.

      I can submit buy, sell, buy, sell orders through Fidelity all day. All the buys will be executed at 25.0599 and the sells at 25.0501. Someone is skimming half-spread. It isn't the guys publicly displaying limit orders; they never get filled.

      So who is skimming the half-spread?

    9. Anonymous2:04 PM

      No offense. But this is exactly why HFT gets such a bad rap. The people complaining do not even understand how the system operates under the hood. It MUST be HFT.

      Look no further than your own Broker.

    10. I don't take any offense if that's actually true.

      But what you are arguing is that Fidelity (and most other brokers) are essentially operating internalized flash systems that skim from their clients. I'd like more documentation to that effect as I don't have time to go full Erin Brockovich.

      I've never had a problem with fast trading, my complaint is with fragmentation, sub-pennying, and lack of a proper queue protocol.

    11. Is the problem payment for order flow? I.e., Fidelity and the exchanges keep 0.0049, reimburse me 0.0001 for my ignorance, all while screwing over the people who post legitimate public at-risk limit orders?

    12. Anonymous3:05 PM

      I can't speak for Fidelity. However, most big brokers either internalize their orders or sell the order flow to a firm that internalizes the orders and sends what it doesn't want to trade out into the market.

      Either way, the orders often do not see the open market. They trade against other clients or they take on the other side themselves if their algorithms see an advantage.

      It is a win/win situation for brokers. They collect a commission from you and they either profit from the trade themselves with no exchange cost to them, or simply sell the flow to someone else.


      I agree that fragmentation sucks (but que jumping is only prevalent in certain circumstances that a normal trader wouldn't be affected by). But that isn't the fault of HFT. And sub-pennying? Most of that is internalization. If you go to an exchange website and look at their order types, you can see exactly what a HFT can and cannot do. And guess what? You can do the exact same thing if you have direct market access (not hard to get).

    13. Thanks, that's helpful.

      I guess my problem is mostly with brokers rather than HFTs. I suspect part of my problem with limits is they get routed to third rate ECNs, and get executed last.

      I need to look into direct market access.

    14. Nathanael11:49 PM

      Aha. You've explained why the market has so much less liquidity than it used to.

      I do standing limit orders, but they're out-of-the-money "let it sit for a month" limit orders. That means I'm basically part of the process setting the ground rules of the market under which all these traders are trading. I expect they'll try to extract money from me, probably successfully, but what are you gonna do; they loot you a lot more on market orders, and there aren't that many types of orders.

  17. I see the only benefit being the massive telecom infrastructure spending in undersea cables like Artic Fibre where Canada basically required that for laying along the NW passage the consortium was required to service much of the small Inuit communities along the route. I love these little externalities. The Norwegian island of Svalbard has some of the best internet access in the world because of the massive excess capacity for the fibre link used by the ESA and NATO sattelite control centers on the island. There has to be some benefit to being trapped in a place where the sun sets for 4 months.

  18. I thought that Fama as father of the EMH suggested on further work that volatility will increase as markets become more efficient. Thus he co-authored the paper that talked about the Tobin Tax with Samuelson (and somebody else?). So I still don't get what "excess volatility" is. Of course you could go the Cochrane route and say volatility is a useless measurement.

  19. Anonymous2:12 PM

    Shouldn't any discussion of the costs of HFT start with what it replaced rather than the pure fantasy of cost-less financial markets? HFTs are simply electronic floor traders. They do the same strategies. The main difference is that now one guy can watch dozens of strategies operating in different markets all over the world. 10 or 20 years ago, you would need dozens of actual humans standing on actual exchange floors to do the same thing. It is bizarre to say HFT is wasteful when it has replaced so many human workers with much cheaper computers.

    Pretty much any game that HFTs play now were more prevalent in the pits. The informational advantages of floor traders were enormous relative to the milliseconds (or really microseconds) that HFTs have. If you were in a trader in a busy pit and good friends with an important broker, you were set. Everyone stood next to each other all day and they were all friends and they all knew that they were in the business of taking money away from the customers. The brokers and the traders worked together to accomplish that goal. Now it is all anonymous and much more competitive. Competing traders aren't friends standing next to each other, they are people sitting at computers in different offices and they don't know each other. That makes it much harder to collude. The result has been an enormous decrease in the profitability of trading. Even though volume has exploded over the last 20 years, trading firms have had to consolidate and cut staff to remain profitable.

    1. Nathanael11:43 PM

      You're confusing HFT with ordinary ("low frequency") electronic trading.

      Ordinary electronic trading -- the original NASDAQ -- was what eliminated the high profiteering of the traditional brokerages.

      HFT, which depends on having your computers very close to the exchange's computers, is a way for the traditional brokerages to get their advantage back and return to profiteering.

  20. Victor G5:26 PM

    I have read your post, professor of finance Noah Smith , to which I say: Sorry dude. You should ether attempt to gain more practical expertise in the field of finance or to try to switch to other academic fields and add more value to the hard science instead of giving bad name to academia by to what Nassim Taleb refers to as “teaching birds how to fly”. Btw, you should read up on his work before arrogantly lecturing practitioners on something you obviously have no expertise to speak of.
    Here’s my advice: maybe you, Noah Smith, should focus on being next Albert Einstein.
    Or at least become a little bit more competent in your chosen field.

  21. VictorG6:28 PM

    Just a little research… 2 .0 out of 5 rating of being professor?
    As former student wrote” PhD in Economics with no real world experience in investments or finance”

    I rest my case.

    1. Anonymous6:51 PM

      VictorG, this 2/5 rating is based on the opinions of 4 students. What would Nassim Taleb say about you resting your case on such thin evidence?

    2. Victor G8:58 PM

      Dear Anonymous,
      Why wouldn’t you ask this Nassim Taleb directly and see what he says. Also, don’t forget to ask him what he thinks about finance being a science. He has some strong opinions on the subject; good luck
      As to my research into life and works of Noah Smith, you are absolutely right, its incomplete. All I did I read a few of his articles, looked up his ratings and stopped on the following gem:
      At the end of the article professor concludes “If America's middle-class and poor people learn to be more cheap, then in 30 years, we will see a very different distribution of wealth.” That statement alone calls for re-taking such advance courses as Arithmetic, Accounting and Taxation but that’s beyond a point of this discussion.
      Also one can’t help but feel sympathy for students. After all, they spend a lot of time and money .Some of them might even feel they been lectured by incompetent idiot but that’s, of course, pure speculation.

    3. Anonymous12:21 AM

      You're a bit pissed at Noah because he's writing about something he has no direct experience in. You may be right. Or maybe Noah does know something that we don't know that actually does pertain to the subject. I don't know. And I don't know Noah, I don't know you, and I don't know Nassim. But I do know that one of the features of the internet is that anybody can spout off about anything they want, and that people spout anger in return. It's all so damn easy to do, but it doesn't help any of us. What do you think, Noah? Does your blog help the world?

    4. Victor G.,

      Oh come off it. If you think that Noah has said something incorrect, then critique that specifically and clearly. Do not waste everybody's time with an ad hominem attack based on a limited set of teaching evaluations at a site known for its small sample bias problems (and of a newly starting teacher at that). So, sure, Noah has not published much, and he might even be a bad teacher. But what does any of this have to do with the quality or lack thereof of the arguments he makes here?

      His blog has become very widely read and influential because he says lots of interesting things on it, often things many people have not thought about, even if he is wrong or misguided in some of his arguments. But, forget about the ad hominem attacks and stick to the ideas, particularly if you are such an expert on finance as you imply. Just what is it he has said here that you find wrong, Mr. Expert Smarty Pants?

      Barkley Rosser

  22. I think you underestimate the costs of HFT. After reading this: I really get the impression we are draining our best minds on a useless endeavor. HFT is all about having a slightly better computer program and slightly better statistical model than all other high frequency traders, which means that they pick our best programmers and statisticians out of fields where they could actually produce some value-added for the economy into one where they have no economic value.

  23. Victor G8:29 PM

    Using Expert Smarty Pants nickname while suggesting “not waste everybody's time with an ad hominem attack “ sounds a little inconsistent , especially from a scholar you imply to be;-)
    But I will answer your question-Noah didn’t say anything.
    Not a single fresh/original thought.
    He quoted someone else ideas, inserted few links and added comments. From the title to the last paragraph, which was used as template for my answer, he proceeds to insult and make fun of profession he apparently knows very little about. Actually, he’s cheering to a fact that HFT making it so tough on humans to compete that they are becoming obsolete.
    In return he tasted some of his own medicine.
    Discussing HFT strategies, its fairness ,compliance with existing rules and regulations and its impact on financial market its beyond scope of this discussion but investigating source-Noah’s- credibility is not.
    I have based my research on publicly available sources and I offered my conclusions.
    Most if not all rating sites have small sample bias problem but that doesn’t mean they should be ignored. For example if 4 out of 5 Yelp reviews from original users called mechanic a fraud, would you take your car there? Why student’s opinion should be treated differently?
    More, I am not pretending to have conducted complete research of Noah’s work although I posted a link to his other article and statements to confirm my evaluation. In fact, one can have a field day discussing ideas presented there. Please enjoy in your spare time.

  24. Anonymous10:56 PM

    There seems to be a lack of understanding about different types of algorithmic trading and HFT. They come in many different shapes and sizes. There is one type that is common to all NYSE listed stocks: Designated Market Maker aka DMM. In the old days, a stock was traded at a specialist station on the floor of the NYSE. One specialist firm was in charge in maintaining a “fair and orderly market.” Specialists went the way of the dinosaurs.

    The specialist function is now assigned to a single company who is the DMM. The DMM is actually a program that runs on co-located computers in Rahway, New Jersey. This is where the “real” NYSE is these days. The old location is essentially a fa├žade used for photo ops. The DMM program is technically NOT HFT, although from a speed standpoint, it is very fast. The DMM program is setup to emulate what specialists used to do. One main difference is that they will NOT take very large positions to hold on “their book” like specialists used to do. Stated differently, they are not willing to risk much of their own capital, be it going long or going short on a stock.

    Totally separate from the DMM algorithm are the different HFT algorithms. HFT’s are trying to extract a half a cent/share or so from each long term buyer/seller. A common form of this algorithm is attempting to predict supply/demand over the next 1 to 10 seconds.

    Arbitrage is another form of HFT. This is where the price on the NYSE is different than the price on one of the other ~40 different exchanges or networks. The HFT tries to make a penny or two on the discrepancy.

    The trades that are done at X.0001 or Y.9999 MUST be done on “Dark pools” because that is the only format that allows “sub penny” trades of less than one cent. Dark pools are typically NOT all that fast and are NOT considered a form of HFT.

    One interesting aspect is that that the largest DMM firms, Getco and Goldman Sachs ALSO have “independent” HFT proprietary trading desks. So they are supposed to be “making the market” on XYZ while at the same time trading for their own account on XYZ. Luckily, these firms are 100% honest and ethical and would NEVER share data between those functions.

    A fair criticism of HFT needs to distinguish between these different types of algorithmic trading. These are others, but these are the main ones.

  25. Nathanael11:41 PM

    Um, a lot of HFT is actually automated front-running. Noah, you need to understand why this is bad.

    And you underestimate the problem of flash crashes. They are EXTREMELY bad. The usual resolution to a "flash crash" is to retroactively cancel a bunch of trades made by real people in good faith. Do you understand what that does to people's trust in the markets?

    This is before we get to the final problem: most HFTs are run by the same old crooked brokerages with the purpose of, as Anonymous says, extracting half a cent a share from each long term trade. In other words, middlemen taking a cut. This can only be done by the very well-connected (literally well-connected, with computers physically very close to the NYSE).

    So HFT is basically a system of extracting wealth from long-term investors (pension funds, etc.) to socially worthless people at Goldman Sachs, JP Morgan Chase, etc