Saturday, April 05, 2014

No one really knows if HFT is good or bad

Like every other human in the known Universe, I'm currently reading Michael Lewis' Flash Boys. But I've been thinking about HFT for quite a while, because Stony Brook has a lot of people who study it for a living (including some people who have done it for a living at Renaissance and other top firms). So even though I'm only halfway through the book - and I'll write a review when I finish it - I have some thoughts on the topic.

Basically, no one knows if HFT is good or bad. There are two reasons for this. The first is that no one really knows what HFT does to markets. The second is that no one really knows what is good or bad for markets.

Why don't we know what HFT does? Well, one reason is that there are a lot of different HFT strategies. Some HFTs could be pure market-makers, just gobbling up bid-ask spreads. Some could be "front-runners", looking for signals of large institutional trades and trading in between the chunks of these trades (this is the strategy Lewis spends a lot of time discussing). There are other common ones (like latency arbitrage, Twitter mining, high-frequency statistical arbitrage etc.). No one has the data to know how much of each of these is being done at any given time.

But a lot of HFTs simply don't know what their strategy really is. They hunt for patterns in prices or orders, find a pattern that seems to work, and trade on it until it stops making money. They don't have any idea why the pattern exists. Sometimes it only exists for a few seconds. In fact, if they stop to gather enough information about the pattern to figure out why it's there, it often disappears! Actually, there are deep mathematical (information-theoretical) reasons to suspect that lots of HFT opportunities can only be exploited by those who are willing to remain forever ignorant about the reason those opportunities exist. It's mind-bending (and incredibly interesting).

Needless to say, we have no idea what this latter type of HFT does to the market, because even its users don't know what it is! But even for the simpler strategies - market-making, front-running, etc. - we don't really know how they impact markets. This is because all of these strategies involve the phenomenon of asymmetric information. Most of our finance theories - the kind where you get nice clean results, like factor models or CAPM-type models - don't involve asymmetric information. Once you add it in, stuff gets complex and weird, fast. If you ever think you understand how financial markets work, go read Asset Pricing Under Asymmetric Information by Markus Brunnermeier, and you will be convinced otherwise.

Do market-makers increase or decrease liquidity? Do front-runners increase or decrease it? What about informational efficiency of prices? What about volatility and other forms of risk, at various time scales? What about total trading costs? Good luck answering any of these questions. Actually, Stony Brook people are working on some of these, as are researchers at a number of other universities, but they are huge questions, and our data sets are incredibly limited (data is expensive, and a lot of stuff, like identities of traders, just isn't recorded). And keep in mind, even if we did know how each of these strategies affected various market outcomes, that wouldn't necessarily tell us how the whole ecosystem of those strategies affects markets - after all, they interact with each other, and these interactions may change as the strategies themselves evolve, or as the number and wealth of the people using each strategy changes. 

Confused yet? OK, it gets worse. Because even if we did know how HFT affects markets, we don't really know if it's good or bad on balance. For example, HFT defenders often say HFT provides "liquidity". Is liquidity good for markets? How much is liquidity worth, are there different kinds of liquidity, and does it matter when the liquidity comes? If I have a bunch of totally random trading, that certainly makes markets liquid, but is that a good thing? Actually, maybe yes! In lots of models of markets, you need random, money-losing "liquidity traders" in order to overcome the adverse selection problem, thus inducing informed traders to trade, and getting them to reveal their information. But HFTs don't lose money, they make money - is their liquidity provision worth the cost?

To know that, even if we knew the impact of HFTs on informational quality of prices, we'd have to know the economic value of informational efficiency. Suppose the true worth of GE stock. according to the best information humanity has available, is $100. Suppose the price is $100.20. How bad is that? How much is it worth, in economic terms, to push the price from $100.20 to $100.00? Is it worth $0.20 per share? It depends on how GE's stock price affects the company's investment decisions. To know that, we need an economic model of corporate decision-making. We have many of these, but we don't have one over-arching one that we know works in all circumstances. Corporations are way more complicated than what you read in your intro corporate finance textbook!

(And this is all without thinking about weird things like behavioral effects of the humans who interact with HFTs...)

To sum up: There is very, very little that we know about HFTs. We have incomplete data, incomplete understanding of the nature of HFT strategies, incomplete understanding about the interactions between ecosystems of strategies, incomplete theories of markets under asymmetric information, and incomplete understanding of the economic value of liquidity and informational efficiency.

So what the heck do we do?? Obviously we need a lot more research, but in the meantime, why not enlist the market itself to help give us some more info? If we legalize and facilitate a number of kinds of alternate kinds of stock exchanges where HFT has a much harder time - e.g. dark pools, batch-auction exchanges, etc. - we can learn a lot (though not nearly all we need to learn) about the overall impact of HFT (and other algorithmic trading!).

Like I said, I haven't finished Flash Boys (and I'll write a review when I do), but my early impression is that Lewis presents a very oversimplified picture of how HFT works - but in plumping for alternate types of exchanges, he gets the solution broadly right.

Here's a similar post by Craig Pirrong, who shares my uncertainty about the costs and benefits of liquidity and informational efficiency, but who is slightly more confident about the net effects of HFT on informational efficiency.

David Glasner has a great post on the topic as well. The biggest cost of HFT is probably the resources that are wasted on "information tournaments". I'm setting up a lab experiment about this very topic, in fact. This problem was pointed out in a famous 1971 paper by Jack Hirshleifer.


  1. Curious - what entices institutional investors to trade on the (new, unproven) alternative exchanges? Are they subsidized in some way so that expected costs are low enough to justify the uncertainty in unexpected costs? It's a fragmented market already, and volumes are down, so are we confident that new venues/models will accrue volume?

    I think many institutions are already pulling back from certain alternative venues in some cases. Whereas the presumption previously was "more venues == more trading = lower costs", now many appear to believe that they are better off in standard venues where the rules are at least somewhat understood, even if they are imperfect. Most people I know seem to think that every new trading model is just another opportunity for someone to reverse engineer the process and trade against you in new ways that you didn't see coming.

    Overall, even with the issues, trading costs are pretty low now, and as long as you're not totally naïve in trading, you can do ok. Good post, though - definitely needs more research.

  2. "Actually, there are deep mathematical (information-theoretical) reasons to suspect that lots of HFT opportunities can only be exploited by those who are willing to remain forever ignorant about the reason those opportunities exist. It's mind-bending (and incredibly interesting)."

    Can you expand on this or provide a reference...?

    1. I can expand on this, though I will have to hunt for the reference (it was explained to me on a whiteboard by another prof). In general it takes more info to understand why a pattern exists than to simply verify that the pattern exists. Therefore if you wait around to gather enough information to figure out why a pattern exists, the pattern may disappear, because someone who didn't care about why the pattern was there can swoop in and trade it out of existence while you're busy figuring out the explanation.

      Does that make sense?

    2. There's a simpler take on this - there are (provably) simple game-like problems and solution strategies where you can always discover the correct answer to the problem in finite time, but you can never prove that your answer to the problem is correct - you can only perform "identification in the limit".

      But whether these problems have anything to do with HFT or asset pricing more generally I have no idea, but if such a pattern existed and agents were willing to trade without proof, then I guess that ti would get arbitraged away very quickly.

    3. Somehow my previous reply disappeared...

      Both of these intuitively make sense. However, I'm something of an information theorist by trade so I'm quite curious about the 'information theoretic' underpinnings of the argument. More, I'm curious to see info theory used in econ (instead of just talking about variance).

      If you have time, I'd appreciate seeing the reference to fulfill my curiousity

    4. Are you saying practically you can never have enough information in real-time to know why the pattern exists (so they could be studied in hindsight and understood, but useless for actual trading), or something more fundamental and brain-twisting, e.g., you cannot gather enough information in these short time spans to know why a pattern exists without disrupting the pattern?

  3. I don't buy this analysis. HFT is extracting a toll of some billions of dollars per year from users of the market, apparently paid equally by buyers and sellers. But clearly the funds extracted should be in the hands of the buyers and sellers, not the HFT traders. By gaming the system, HFT traders have effectively erected these toll booths which no one can avoid. Do the toll booths add value in some other way? Seems to me the burden is not to prove that they don't, but to prove that they do. Ask the big volume buyers and sellers (mutual funds and pensions) if they are happy paying the toll. They may not understand the mechanics of how it is being extracted, but I am sure they would be happy not to pay it. For retail and small volume investors, the toll is so small it is irrelevant; in fact, even for huge institutions it probably amounts to only a few bps. But those bps belong to the investors. My view: HFTs are pond scum; very clever, very agile and very rich pond scum.

    1. Of course, David, if you describe HFT is such tendentious terms, you will evaluate it negatively. But your terms have no basis in reality: where in the world have HFTs "set up tolls"? Who is being forced to pay them anything? All they are doing is buying stocks they think will go up, and selling those they think will go down, *just like pretty much every other trader in the world*. But for you, somehow they very *speed* with which they do this makes them "pond scum." So if you buy a stock today you think is underpriced and sell it in a year for a profit, you have not "set up a toll" and your profit should not "clearly be in the hands" of someone else: why not? Why is it morally different than me doing the same thing over the period of one minute?! If the activity is wrong, it is wrong, however slowly one does it!

    2. As you may know, there are other kinds of markets to discover prices and provide liquidity, besides brokered markets. I suppose all have their pros and cons, what is most suitable.

      Yes, all brokers get a fee. All dealers get a fee (or compensation). All market makers get a fee. (Depending on the rules, all three of those things are not necessarily the same.) You might conceptualize that none of these fees are worth it, but that is debatable, not obvious.

      In the financial markets, back in the 70s, it was decided that the fees would be negotiated.

      It seems that we are now at the long end of that, with "fee" being re-written in new and interesting ways.

    3. Isn't the lack of large outcry from large pension and mutual funds about HFT the dog that didn't bark? Those groups have tremendous power and on balance don't seem too concerned with HFT. If anything, the concern seems to stem from other market makers competing with HFT.

      Here is Cliff Asness of AQR on HFT (they manage $100 billion):

      "How do we feel about high-frequency trading? We think it helps us. It seems to have reduced our costs and may enable us to manage more investment dollars. We can’t be 100% sure. Maybe something other than HFT is responsible for the reduction in costs we’ve seen since HFT has risen to prominence, like maybe even our own efforts to improve. But we devote a lot of effort to understanding our trading costs, and our opinion, derived through quantitative and qualitative analysis, is that on the whole high-frequency traders have lowered costs."

  4. Noah you said, in illuminating fashion....."In fact, if they stop to gather enough information about the pattern to figure out why it's there, it often disappears! Actually, there are deep mathematical (information-theoretical) reasons to suspect that lots of HFT opportunities can only be exploited by those who are willing to remain forever ignorant about the reason those opportunities exist. It's mind-bending (and incredibly interesting)."

    Your description above reminded me a lot of how I see macro.

    Could HFT be a way to look at economics in a microcosm ? If there were analogous situations in the two realms, then we could observe outcomes that would take a lifetime in macro happen in moments in HFT. The programing in HFT is about making money, but what if it could be directed instead to test a hypothesis ? It's next to impossible to run real world experiments with macro ideas on a macro level. Could HFT be a window to macro economists being able to run lab experiments ?

    Probably not...LOL.
    Anyway Great post. I am reading Piketty's "Capital in the Twenty First Century" right now and this dovetails nicely.

  5. Are you saying "no one really knows" if (Lewis' thesis) the market is rigged by HF traders? Or perhaps, no one really knows if a financial system being skimmed by a small group of HF traders is good or bad? Is it possible to lean too heavily on uncertainty?

    1. I'm saying both of those things.

    2. What would be the possible upside to skimming?

    3. HFT skimming might be cheaper than broker/dealer skimming, and might drive broker/dealers out of the market and replace them.

    4. By far the best post I've read so far on the subject. You've got deep insights into how it really is, spookily congruent with what we practitioners experience.

      One of Lewis's brilliancies in promoting his book has been getting everyone presumptively closed on "skimming" and "frontrunning". I was glad to see you put it in quotes. The description of frontrunning in the book is factually wrong. There is no such profitable trade as "buy in front of an unexecuted seen order, raise price by some undefined process, sell back to original order at a higher price". What happens is, a market maker sees information leakage from a below average trader, someone like Brad in the book, and is immediately alerted that he has just sold a large chunk of his own inventory, on Bats, at too low a price. He immediately races to cancel all his other same-priced offers on all the other exchanges he's quoting. This is a survival behavior, and the market maker would not be in business if he didn't adjust to such information. Once Brad hired Ronan the network guy, and became an above average trader, he was able to execute all his orders, not just the first one, at prices unjustified given the size he had to trade, were it known to the market. Thus, Brad became the skimmer, and any retail traders on the other side of his trades received too little for what they sold, given the true impact of Brad's now hidden private inside information about how much he wanted to buy. HFT mitigates this phenomenon to the extent possible, but it's certainly legal for traders such as Brad to hide their own private inside information. I don't understand why Lewis chose to view HFT in such an asymmetric manner, it's as if he doesn't care about the fairness to the other side of the trade when large size is trying to sneak through the market. It's a highly anti-retail investor stance, but I can't tell whether he's being disingenuous or whether he simply doesn't understand this.

    5. Well, let's hope we all don't end up in a legalese dead-end on "front running" and "skimming". God knows, this "it was all legal" meme is certainly popping up, maybe interrupting the real conversation about market integrity.

      Elsewhere I talked about everyone conceptually trying to come to grips with the pitfalls of speed (yes, pitfalls - it's not all goodness).

      In the situation you describe, making use of a high-theory name "information leakage", what is happening is that speed has turned the market maker, usually price taker, into a price maker.

      This interrupts our basic sense of fairness. It just does. To illustrate, you show up at the counter to check-out with your clothes, and the check-out clerk looks at all you have and says, "okay the first three are at the marked price and the next few, well, I know we said we'd sell them to you at the marked price, but we'd like a little more please, not too much, just let's split the difference between the price we offered to sell them and what you can get across the street at another retailer, okay? I know you don't like that, but, trust me, it's really better for everyone this way, especially those who come through with just one or two things to buy."

      Speed also has an impact on our concepts in other ways. For example, true or false: superior speed plus "information leakage" has the net result of turning a market order into a limit order of sorts, which can in turn be conceptualized as an option. That is, speed (or certain information advantages) can turn an order into something like an option.

      I have to think about that some more, but it's food for thought, no?

    6. Nathanael11:52 PM

      It's worth noting that the obnoxious and illegal front-running skimming by HFTs is widely considered to be one of the reasons why many traders are dropping out of the market entirely and heading to harder-to-scam fields such as direct purchase of land.

  6. I suspect that having a multitude of markets will not help ordinary participants unless the competition between markets over trading fees drives those fees down. Having a multitude of markets will give rise to arbitrage opportunities between those markets.

    Most market participants would probably be better served by a system where one central system took in offers to buy and sell and matched them on a first come first served basis (the first offer at a given price is the first filled) with no offers more complicated than a limit order being allowed. Offers should be required to remain open for some minimum period of time (like ten seconds).

  7. Noah, I'm truly not being a wiseguy (for once): In order to understand your claims about our ignorance concerning HFT, can you give really short statements about other things? E.g. do we know if futures markets are a good thing? What about really sophisticated options? What about regulations against insider trading? What about farming?

    1. I thought a wiseguy was someone in the mafia?

      I like farming. I have a thing for farmer girls.

    2. But you can't really predict how they'll interact.

    3. That's what creates the value. ;-)

    4. That you can't predict how farmer girls interact is one of their charms.

    5. Exactly. Silly Austrians FAIL TO UNDERSTAND THIS

  8. Legalize insider trading until Stonybrook can complete a study determining its costs and benefits.

    For that matter, legalize murder while we await the results of the study.


  9. Try trading stocks!

    I get executions for 3 shares on 500 share orders, and NEVER get to sell shares on a limit order unless the stock is going to rocket through my limit. If not, the machines trade vs my orders, even if they are small. I daily see order executions made at .0001 below my 500 share offering. In stocks with middling liquidity you have to be an idiot to place any material size limit order. As a consequence markets are thin.

    This is different than what specialists used to do 20 years ago. The difference is that my orders are handcuffed by only being able to offer in even cent increments of price, while the HFT's can undercut my offers out to infinite decimal points.This minuscule benefit to the counterparty only occurs because someone else with real investment intent is backstopping the HFT. Specialists at least had to split sales with the book, and couldn't step in front of retail orders.

    The other difference is that the HFT has ZERO obligation to maintain a continuous market. Specialists were required to, and it was sometimes extremely costly.

    What is the same, and will always be, is that better placed traders take advantage of the less well placed traders. It's like the old Spy vs. Spy cartoons, where reactions to gaming the system create other gaming opportunities. I find it annoying, but what else is new?

  10. HFT and algorithmic trading go hand in hand. Most HFTs are driven by algorithms. HFTs create inefficient trading market that last for milliseconds and algorithms exploit at the expense of almost all traders including hedge funds.

    Another way to look at it is this: only a fool would spend hundreds of millions of dollars in hardware, software and algo team iff HFTs did not make money.

    1. Meh, close but no cigar. All HFTrades are algorithmic but not all algorithmic trades are HF.

    2. Early algorithms were constrained by inability to trade fast, then faster, then fasterer! HFT was the techno solution to initially support the algorithms, but now these two are so badly entangled that one can not separate.

      Buy and hold is an algorithm that does not need HFT!

  11. Noah,
    This was an excellent, very informative post. I look forward to reading your review of Lewis's book.

  12. If the chart to scalar theory is correct, HFT would make the market more stable

    The equation for standard deviation

    if the trade size a_v is small relative to v_r then the standard deviation of moves is smaller. On the other hand, if we make a_v, v_r,and v_0 increase by a scaling factor it cancels out and sigma is unchanged. So in the context of this model, HFT is either stabilizing or neutral.

    For example before HFT let's assume we have: v_l=10^6, v_r=10^6,a_v=10^3

    now introduce HFT: v_l=10^7, v_r=10^7,a_v=10^3

    we've increased the volume of the stock on the LHS (left hand side) resistor and the RHS but the trade size is unchanged.


  13. Noah, is Brunnermeier's book accessible to a general audience?

    1. It's really just a reference to a bunch of academic papers. You might enjoy it though.

  14. I originally left this post at Marginal Revolution, commenting on Tyler linking here. Overall much better treatment of HFT than 95% of what's out there, but a few criticisms:

    “They hunt for patterns in prices or orders, find a pattern that seems to work, and trade on it until it stops making money. They don’t have any idea why the pattern exists. Sometimes it only exists for a few seconds. In fact, if they stop to gather enough information about the pattern to figure out why it’s there, it often disappears!”

    Noah’s making it sound like since automated systems need to execute quickly that implies the researchers have no time to reflect. One has nothing to do with the other. Models can be carefully developed over months or years, carefully studied and understood. Yet they can still require millisecond execution to be profitable. How long the typical trading opportunity in a strategy has nothing to do with how long a strategy remains profitable.

    “Actually, there are deep mathematical (information-theoretical) reasons to suspect that lots of HFT opportunities can only be exploited by those who are willing to remain forever ignorant about the reason those opportunities exist. It’s mind-bending (and incredibly interesting).”

    Huh? I have no idea what this is referring to.

    “Actually, maybe yes! In lots of models of markets, you need random, money-losing “liquidity traders” in order to overcome the adverse selection problem, thus inducing informed traders to trade, and getting them to reveal their information. But HFTs don’t lose money, they make money – is their liquidity provision worth the cost?”

    Noah seems confused, the “liquidity traders” in the models he links to are liquidity *consumers*. Liquidity providers, e.g. HFT or specialists, play an entirely separate role. Liquidity providers earn non-negative profits under any non-absurd set of assumptions.

    “It depends on how GE’s stock price affects the company’s investment decisions. To know that, we need an economic model of corporate decision-making.”

    If this is the reason to agnostic about HFT, then it’s also a reason to be agnostic about literally all market economic activity. Noah’s saying since we don’t know how exactly entities respond to price, we can say nothing about the utility of accurate prices. How many economists are “agnostic” about farm subsidies? After all we don’t have an exact model about how food consumers respond to the price of food. Would Noah say that he’s unconvinced then that corn subsidies are economically inefficient?

    Despite those criticisms I essentially agree with what Noah’s saying. (Though I would say the balance of evidence and theory is much more tilted towards the positive side than agnosticism describes.) When faced with deep uncertainties about why institutions or activities exist in human society one should defer to Hayek. There are many systems that are products of human action but not of human design. Attempts to monkey around through central planning on these systems often leads to unexpected, deep and destructive side effects. Promoting market alternatives is a far less risky approach than taking a stab in the dark with poorly understood regulation.

  15. Well, rather than a retreat into darkness, perhaps one can apply some decision making or design under uncertainty.

    Why not just focus the lion's share of light shedding on market-maker activities, on the goal of having competitive market makers and the role of electronics and HFT in the market-making process, with both its good aspects and pitfalls (including speed and speed-pitfalls, like speed differentials)?

    To wit, look, if we don't know and cannot tell if high-speed electronic trading _for the purpose of market making_ is less frictional and more liquid than slower, manual alternatives, than WTF? Seriously, the whole promise of ECNs and machines is that they help us to be more productive. If we cannot tell how or if that is occurring, then we're simply doped.

    If the market, left to its own, is "fragmenting" or "specializing" (by platform or by order size or perhaps even order type), and we have no objective utility measure for that, then who is to say we are moving toward optimality and not toward chaos? Why aren't the enormous scale advantages of electronic trading drawing more and more to a single platform? If we we have theoretical reasons to think that the small guy is getting a ride, yet have no measure of that, then our precepts of microstructure/market structure are more like diplomacy than empirical science.

    If we cannot say whether platform incentives (negotiated or given in secret?) to electronic market-makers are distorting prices or generating (a) unfairness or (b) perceptions of unfairness than we've lost a lot more than we've gained, no matter how much people excuse by talking about oligarchs, helping the little guy in a free rider kind of way, or the bid-ask spread, etc.

    Now, after one's answered or considered just the basics of market-making, you know, ready to buy-and-sell at a given price with your own ("informed"?) capital and smarts, consider the dynamics of that with high-speed trading stratgies.

    Are we really so helpless as to not know? I mean, what types of trading strategies interact "deliberately" with market making activites in such a way that we scratch our heads? Can you do things that "trade your own book" in a sense, can you stuff quotes, can you blink and paint the tape, can you lift your own offer to avoid having an order filled, just because you are "faster"? Have you set off a silly arms race to be first-past-the-post and the next order "node" (exchange), when it would be more efficient for same time arrival (and/or a decent bid-offer), either on an individual basis or on a wide, social, capital-allocation basis (re: why does society benefit by laying fiber under the artic)? Have you tilted the playing field to advantage large, well-capitalized firms, but not too much, so that, you know, real and full competition doesn't dry up down the road? Is trading "anonymous" or can you figure it out?


    1. "If the market, left to its own, is "fragmenting" or "specializing" (by platform or by order size or perhaps even order type), and we have no objective utility measure for that, then who is to say we are moving toward optimality and not toward chaos? Why aren't the enormous scale advantages of electronic trading drawing more and more to a single platform?"

      What you call fragmentation could also be competition. The increased fragmentation of equity liquidity is largely the result of new offerings, BATS and DirectEdge, taking market share from entrenched players, NYSE and NASDAQ, because they offered a better, faster and cheaper product.

      There are markets where all trading occurs on a single exchange. Primarily futures, due to their centrally cleared nature. If you think this represents an improvement over the equity markets you haven't looked into the much higher fees and crappier infrastructure that exist in the futures monopolies.

    2. When you do not observe the results you expect from *enormous* scale economies, like electronic trading, it bears further investigation, no?

      Here is the best graphic I've see so far on order execution that also gives a picture of market structure / topology:

      I'm not going to take the conversation too far afield, although this original post is highly theoretical in its tenor. To that end, why do so many assume that competition is a good thing for every circumstance?

      Afterall, pure competition leads to zero economic profits and, quite often, creates quality offerings so low, in some parts of the markets or more widespread, that we recoil at them and their practices.

      And there are plenty of unfair competitive practices, like dumping and buying market share. Could an upstart exchange compete unfairly to get market share?

      In the digital age, information is incredibly valuable, information about market participants and their holdings and behaviors. Let's say it is so valuable that bid-ask is no longer a meaningful measure of anything, just a residual of a days-gone-by approach to selling, dealing, brokering a market for financial stocks. Let's theorize that, like other enterprises, there is and will be a consolidation of the economic "power of information" into the hands of a few that has vast implications for how we trade and perceive the fairness of trade way down the road (not just stocks, but all economic transactions, including such practices as first-degree price discrimination in a market for goods and services).

      Against that consideration of a march toward 'information consolidation', what does it mean, in the digital age, to create an "informationally competitive market"? Can it be done?

  16. [cont]

    Finally, we can look at high-speed "quant" stratgies that are NOT designed to deliberately interact with market making and do so incidentally, as a matter of course only. One interesting exmaple, Noah points out might be finding and trading random correlations until they break down, in one way or another, either the profit gets too thin or someone figures out the fundamentals of them and, in becoming self-aware, makes them go away.

    As we all know (not in the dark), the "policy" risk with these is that they become self-reinforcing and lead to crashes or freezes in the market as everyone tries to do the same thing at once, right? That kind of stuff is pretty well studied and even if there is not a Ministry of Approved Strategies [although I think we'd all like to be the Minister, if it paid well], we think we know ways to circuit break and so forth (capital requirements, other rules) to deal with the consequences of at least that kind of trading, both at a firm level and at the exchange level.

    edits: in the prior post "getting a ride" s/b "getting a beneficial or 'free' ride"

  17. Considering what recent history has been like, all things of this kind are guilty until proven innocent.

    And what is with requiring a Google account in order to comment? Economist's View manages without that; why can't you?

    1. I noticed an increase in the number of spambots and temporarily turned up the filter.

  18. Just do the experiment. Try various ideas for eliminating types of HFT. Turn them on and off a bunch (like say 1000 times a day, heh, HFT experiment). See what happens. It shouldn't be dramatic given that nobody can tell anything much has changed as they were invented.

  19. I think I'd get a bit more out of this post if I knew what an HFT is.

    1. It's the Hartford Federation of Teachers.

    2. Nathanael11:53 PM

      High Frequency Trading, also known as High Speed Trading.

      It's a scheme for front-running orders by intercepting other people's orders before they hit the market.

  20. Correction: "To sum up: There is very, very little that we know about HFTs" should read: "To sum up: there is very, very little that I, Noah Smith, know about HFT."

    Thank you for your attention.

  21. Beat the market - let your house cat invest?

    about naive "algorithmic" trading.

  22. I came to economics with a humanities background, so forgive me, but.... it continues to amaze me how much brain power is being spent on making money out of nothing. If Michael Lewis is oversimplifying HFT, then this post seems to overcomplicate it. I can find no benefit in traders acting more on instinct or "patterns" they might see but don't understand or in holding positions for just a couple of seconds or even less. Long-term investors, or value investors believe in their investment, that's why it's called investment... HFT is just exploiting arbitrage opportunities. About liquidity... on balance, liquidity increases prices, at least central banks seem to think so. I do think that higher prices indicate the health of an economy, provided that this implies that society in general is wealthier. But I'm not sure if this is the case with HFT which is more a tool of already rich people to get even richer.
    There are "simple" reasons why HFT might be bad, but there are no simple reasons why HFT might be good.

  23. Do you have further information on the existence of "deep mathematical (information-theoretical) reasons to suspect that lots of HFT opportunities can only be exploited by those who are willing to remain forever ignorant about the reason those"?

    1. I do not, since it comes from lectures given by a prof here at Stony Brook. But I will try to get you some better info.

  24. If there is a social good in HFT, don't you think that HFT would be able to express it to defend themselves?

    Parasites may do some good. Is that reason to tolerate parasites?

    1. Yes it is. There is some study out there that the explosion of allergies out there is directly related to our ability to eliminate parasites in the developed world. The argument goes that humans evolved their immune systems to deal with parasites. Most parasites suppress immune response and humans evolved greater immune responses to counteract the effect. If you eliminate the parasites that basically all humans had you are left with a very reactive immune system that now reacts to non-dangerous environmental factors. It may be better to actually have certain parasites in the long run because in responding our body is actually in a symbiotic equilibrium. Of course this is all in balance because severe infections are certainly deadly especially with other pathogens. The same could be suggested of the markets. Sometimes knowing a little is worse than ignorance. When western doctors started to give iron supplaments to tropical populations to allieviate endemic anemia it worked. It also made them fantastic hosts for malaria and caused tens of thousands of deaths more than the anemia ever did although most anemia deaths are horrific in that they are often women in childbirth.

    2. Nathanael11:46 PM

      In contrast to the benefits from parasites, there is no benefit to HST.

  25. Hi Noah,
    You are most likely quite right about the observation that nobody really understands what HFT is all about. Including the HFT guys.

    I get the feeling sometimes that the closest analogy that I can come up with, is the fictional (?) thief who manages to clip 1c from everybody's regular paycheck and goes un-noticed. For a while anyway.

    1. meh

      After a week of poking around with this stuff, here's an alternative view:

      The key parts of "HFT" insofar as they are related to market-making events are pretty simple. In fact, if you understood how CDOs got made and leveraged, when that longstanding opacity was finally widely laid bare after the fact (to all our economic chagrin...), it's probably a LOT simpler than that.

      Yeah, the mathematics might be complicated, but the basic intuition about what is going on is simple.

      The real "complexity" comes in other ways, mostly non-theoretical practitioner-savvy ways, such as the implementation the algo across multiple markets for the same stock (latency arbitrage and so forth) and in the inducements and quiet facilitations from "friendly" exchanges and brokers, and in the ancillary/supporting strategies (quote stuffing, tape painting, searching, purchase and segregation of order flow).

      Of course, the computer skills at this level of desired speed and volume are sizable, but those seem to be a specialty, a complexity that is not required for full understanding, apart from savvy risk control.


  26. about that hope for more research:

    from Flashboys footnote:

    "In March 2013, the Commodity Futures Trading Commission, a derivatives regulator, ended its nascent program to give outside researchers access to market data after one of those researchers, Adam Clark-Joseph, of Harvard University, used the data to study the tactics of high-frequency traders. The commission shut down the research after lawyers for the Chicago Mercantile Exchange wrote the regulators a letter arguing that the data Clark-Joseph had collected belonged to the high-frequency traders, and that sharing it was illegal. Before he was booted out of the place, Clark-Joseph showed how HFT firms were able to predict price moves by using small loss-making stock market orders to glean information from other investors. They then used that information to place much bigger orders, the gains from which more than compensated for the losses."

    1. This is the benefit that HFT are suggesting. They are forcing investors to participate more often and thus reveal their information to the market sooner and in smaller amounts. As the information is integrated into the price by inducing the trade the HFT take a cut of the difference to compensate them or their services.

    2. Thanks for your comments!

      Three things come to mind.

      1. We've had a long tradition of letting the markets be studied, at least the public exchanges. It seems risky to give that up entirely. After all, we've uncovered mischief by doing it. The NASDAQ bid-ask bru-ha-ha comes to mind.

      2. We ought to have an objective measure of the cost of the type of price discovery you describe. The word "sooner" in your statement means everything when it comes to conceptualizing the impact of speed on ... "fair trade". Does it mean blaze around and buy up liquidity so it can be sold back to a seeker of it? Does it mean move your mkt-maker price up higher and faster than others in anticipation of a big buy order? Does it mean using sly order types to limit your risk of adverse selection?

      3. Also, we should look at the cost of smaller amounts, in another way, because it seems to have lead to market fragmentation, possibly. One impact seems to be flight of larger players as they figure out that they'd rather be at the proverbial high-stakes table, the one where buyers and sellers are pre-qualified, so that they know they can sell 250,000 shares at a clip, if there is a buyer.

  27. I highly encourage you to think about this in a different dimension: Market making and liquidity pools are natural monopolies. Traders tend to gravitate to deep liquidity pools. Kinda like Facebook - who wants to be the only guy on Facebook? Similarly, nobody wants to be the only one in a market. The bigger you are (e.g. Fidelity) the more you value deep markets. Once a market maker has a critical mass of liquidity and customers, it is very hard to knock them down. Very high barriers to entry.

    It does not matter whether the expenditure is "wasteful" - what matters is who accrues the benefit of innovation. Does it reinforce the natural oligopolistic structure of the market, or does it break it down and make it more competitive? "front running" by a computer program distributes demand across several pools of liquidity, reducing any particular market makers "hold" on a critical mass of liquidity. Is it easy or hard to enter the market with a new strategy?

    "Volatility" in individual names is driven in part by sudden demand facing a fixed short term supply curve. "Volatility" in something like the S&P index is driven by macro factors. In my simple mind you can boil down informational asymmetry and diffusion into the elasticity of the short term supply curve.

    reinforcement of high barriers to entry and an oligopolistic structure will increase volatility by decreasing the elasticity. Decreasing barriers makes the supply curve more elastic.

    All this pursuit of this or that HFT strategy is interesting, but seems to miss the salient point that market making itself is a natural monopoly. I do not know if HFT is good or bad because no one has told me the effect on market structure and who is accruing the benefits. Finance people make a lot of money off rents. How is this affecting the ability of finance people to collect rents in market making?

    1. It's an interesting and good point about natural monopolies.

      Of course, there has been off-the-floor trading in stocks, for a variety of reasons, that gives a lot of market power to the big brokers/banks.

      Nevertheless, if someone had said "competitive market makers" 20 years ago, I'd guess that most would have thought most easily of more competition on a single platform or two, rather than a proliferation of exchange venues / definitions that we've seen.

      Although perhaps supportive of HFT, one doesn't need to be agnostic. There are and can be trading that we'd classify as manipulative market practice. If I'm not mistaken, there was a time when 'cornering the market' was not illegal, no?

      My personal read is that a number of people thought that automating trading would lead to a cleaner market than the old-boy way. It looks like it has been a wake up call to the expectations of some and the conscience of others to find that "trading" can bring out the worst in people. Or, if that is too harsh, make them short-sighted in any number of ways.

  28. This comment has been removed by the author.

  29. I agree with the position that the burden of proof is on the HFT folks. From what I gather from the press and the blogosphere, many of the various mechanisms under the general rubric of HFT have zero transparency. Isn't a key feature of an exchange that the rules are clear to all participants? Lewis indicates that even major retail brokers like Vanguard and Fidelity were in the dark. If this is a benign or advantageous feature, wouldn't the stock exchanges be advertising it to all users? If we really don't know how it works, how do we ensure that market stability will be maintained, a scant seven years after the last crash? Sorry, not buying it.

    (reposting after minor editing)

  30. No, the burden of proof is absolutely *not* on HFT guys. There is no evidence *at all* to think that HFT is bad, parasitic, or unfair. The criticisms that unprincipled dicks like Lewis are spewing are nonsense.

  31. Nathanael11:45 PM

    Most of HST is front-running.

    Front-running is illegal and has already been proven to be bad.

    Therefore most of HST is bad.


  32. I have lot of qualitative posts at your blog, and this being one of them .

    Thin Client Hardware & Zero Client