Wednesday, April 16, 2014

Guest Post: Still confused about high-frequency trading? Yes.

Noah's note: This is a guest post by Zachary David, an HFT practitioner. You can follow him on Twitter here. My view of HFT is not quite as benign as his, but that's probably to be expected. I haven't seen many actual practitioners in the big media discussion about HFT, so I thought I'd get one to offer his own perspective.  

I’ve felt like a popular boy lately. In the five years I’ve been developer and researcher at a small trading firm, my family and friends mostly thought I just look at numbers, write code, and play with computers all day; they thought mostly correctly. But Michael Lewis’ recent indictment of high-frequency trading (HFT) changed all that. Now the mystique of “the black box” pervades the blogosphere and news media. I’m often asked for my thoughts about so-and-so’s piece or if regulation is going to kill my job. The discerning readers of Noahpinion wont be surprised when I say that most reporting on this topic falls somewhere between incomplete and wrong.

Today Evan Soltas published an introduction to high-frequency trading on Ezra Klein’s new Vox Media: Confused about high-frequency trading? Here’s a guide. But rather than a guide to HFT, it’s only a guide to the narrow set of topics made popular by Lewis’ book — which pertain only to US equities are not representative of the whole field. While this critique could easily be applied to most of the recent HFT coverage, Mr. Soltas is a very clear writer and that makes it easy to do a point-by-point analysis. Each of the subject headings below corresponds to the respective heading in the Vox piece.

(As I preface all of my pieces: there is a non-zero probability that I err. My focus as a developer has been primarily in futures and currency markets. While currencies are fragmented similar to US equities, there are specific nuances and regulations of which I do not have experience)

What is high-frequency trading?

 ... High-frequency trading is a kind of market activity that moves in less than one millisecond to spot and take advantage of an opportunity to buy or sell. It happens through trading algorithms, programs that determine how to trade based on fast-moving market data. ...

In this section we learn that HFT is faster than blinking, what type of market activity it analyzes, and a type of trade it might make. But we don't get a precise definition of what it is. Allow me.

high-frequency trading: the ability to quickly execute trades and manage order activity.

This is a broad definition because the field is broad. Loosely, the three main categories of automated trading strategies are (1) latency arbitrage, (2) market making, and (3) statistical arbitrage. There are many algorithmic firms employing many different strategies that fall into some combination of the above. However, it is important to note that none of these strategies are risk free and thus arbitrage, by the economic definition, is a misnomer. Unfortunately, Mr. Lewis has pigeonholed the entire industry into number one. And number one is very boring.

(This piece is going to be too long already. So I wont get to talk about the cool fun stuff like market making and non-latency sensitive stat arb. Perhaps another time)

Is high-frequency trading growing?

Not anymore, according to most data. High-frequency trading came into vogue during the 2000s, but after many traders entered the market, profits are way down, and there seems to be slightly less high-frequency trading than there used to be ...

Here Soltas conflates the decline in firm profits with the incidence of HFT itself. This is odd because in economic terms, a perfectly competitive market implies that long term profits will be zero. This is exactly what we would expect to see as competition increases.

Generally, academic research pertaining to HFT firms uses the SEC (2010) definition:

HFT firm: market participants that end the day with close to zero inventories, frequently submit and cancel limit orders, use co-location facilities and highly efficient algorithms, and have short holding periods.

While this is a good definition for HFT-only firms, the declining costs of co-located servers, declining costs of latency-based solutions from software and hardware manufacturers, and the increase in the amount of open source projects has broadened the ability for any firm to do HFT-style execution. The industry magazine Automated Trader frequently covers the fast execution algorithms which have allowed firms to get into large positions with minimal impact on market prices — meaning less opportunities for other firms to capitalize on. (Last I heard, the market rate for a custom high-frequency platform is $2-4 million)

So yes, high-frequency trading is growing even if HFT-only firm profits are decreasing.

How does high-frequency trading make money?

... When the traders see CalPERS place a bid for Apple shares on the tech-heavy Nasdaq exchange, they quickly buy shares on other exchanges, inferring that CalPERS' orders are coming down the wires. Then the high-frequency traders sell the Apple shares back to CalPERS at a higher price than they paid for them a millisecond ago. This "electronic front-running" happens because the high-frequency traders have an advantage in terms of speed, and because "the stock market" doesn't really exist — what exists are many stock exchanges in a trading network. ... 
... The third exploits the network structure of markets, and the fact that they don't all adjust instantly to changes in price. If high-frequency traders can figure out where a stock price will be in the next millisecond before other investors can get a quote, that's a huge advantage they can use for profit.

Yes and no. Yes, that is roughly the trade Mr. Lewis describes in his book. But the reality is a bit more nuanced. First, I think it's unlikely that CalPERS has a Market Participant Identification (MPID) on the NASDAQ OMX, so these traders wouldn't identify the orders as belonging to CalPERS (from their website it looks like they conduct their trading through various brokers).

More importantly, neither of those trades are risk free. In the first case, firms are making bets that someone is trying to get into a large position and will pay whatever price is offered. In the second, they are betting that the price change they witnessed on one exchange is actually where the market is going to be. 
Here's an example:

two different markets trading the same stock
Imagine these are the order books (i.e. the sets of bids and offers by price and size) for the same stock on two different markets. The HFT firm sees someone place a buy order for 2 on Market A at 16127 (making that price the new best bid with a quantity of 1). So the HFT firm does the same on Market B. Now in order to profit from that movement, they have to be able to sell it at a higher price. If someone else comes in and starts selling, forcing the price downward, the HFT firm will lose money on that bet.

Can high-frequency trading cause stock-market crashes?

The high-frequency trading algorithms simply move too fast for humans to intervene with better judgment. When stocks drop, the trading programs may decide to stop trading, withdrawing liquidity from the market, or they may add to the sell-off.
This supposes that humans do have better judgement. But market crashes have happened long before the existence of HFT. In a previous post, I noted that even completely automated firms have people, called operations or “ops”, who monitor the overall health of the systems to make sure they are functioning as desired. When something out of the ordinary happens, they have to quickly figure out whether or not to turn off certain algorithms. No one flips the switch and shuts their eyes (except some people working night desk, *cough* *cough*).

That wouldn't surprise many people who remember what happened to the stock market on May 6, 2010 at 2:45 p.m. — the "Flash Crash," in which U.S. stocks fell 9 percent and then recovered in the course of a few minutes. Shares in companies like Accenture, a management consultancy, fell from $40 a share to a penny.

Oh the good ole Flash Crash. Amid all of the phenomenology reports and finger pointing, it's easy to forget the general conditions surrounding the event. There was massive political uncertainty and the markets were already volatile. All morning, CNBC had a picture-in-picture box of the riots in Greece. Further, it's a bit disingenuous to say share prices fell to a penny. All that implies is that there were no orders in the order book — very little trading actually took place at the low prices. During an equivalent panic in the days of floor traders, no one would raise a hand to trade.

What caused the overloading, Nanex argues, was "quote stuffing" — high-frequency traders that sent in a blizzard of orders to buy and sell at the same time, only to cancel those orders milliseconds later before they went through.

I'm highly skeptical of Nanex's data and methodologies. Like all of their claims, the "quote stuffing" scenario has never been validated. Recent data suggests Nanex's research is fraudulent. Analysis that comes from them should not be treated like peer-reviewed research.

Are there other possible problems with high-frequency trading?

1) Much high-frequency trading exploits data before it is public for an advantage.
Until last summer, the data firm Thomson Reuters, for example, sold to elite investors the right to see an important economic statistic, the University of Michigan's consumer confidence survey, five minutes earlier than the rest of the market. An "even-more elite" group of high-frequency trading clients could purchase an extra 500 millisecond head start.

Important economic statistics are released on regularly scheduled dates and times. Market participants know these well in advance. If a firm conducts trading that is sensitive to an economic number, the firm appropriately removes bids and offers from the market, and possibly closes its position, prior to the number's release. There's very little advantage to be had.

2) Some strategies in high-frequency trading, such as "pinging" and "spoofing," are unethical or illegal.
"Spoofing" is a strategy, ostensibly banned in 2010, in which high-frequency traders send in orders with the idea of trying to confuse, or "spoof," other traders - and especially other trading algorithms - into thinking that demand to buy or sell a stock is coming. If the other traders fall for it, the algorithm quickly reverses course to take the side of the trade it actually wanted. There's evidence that this is what trading algorithms sending in bizarre orders, as they did during the Flash Crash, might be up to.

Again, I need to reiterate the fact that these are all unverified claims from a very likely fraudulent researcher.

What are some ways we could curb high-frequency trading?

I wont use Noahpinion as a soapbox to discuss why all of the proposed ideas are awful, or the endless amount of unintended consequences that comes with any idea "to redesign the way markets work."

But I will say Finem Respice — look to the end. There are many electronic markets in the world that trade under many different sets of rules. We should seek solutions that are already in use in real markets. The currency exchange EBS has successfully experimented with several ideas. First, after they increased the minimum price increment (tick size) of several currency pairs, they noticed more liquidity at every price level. More recently, they've introduced latency floors which randomize a queue of incoming messages in several millisecond buckets. That has also been a success and they are expanding it to all of their products.

Thank You

Thank you Noahpinion readers! (and thank you Noah) I appreciate the opportunity to talk about my job a little bit before it becomes socially cold for another 5 years. And unlike my blog, people actually read this one. So I am greatly looking forward to the excellent commentary that typically accompanies a Noah post.


  1. Anonymous1:40 PM

    Great Post, I'll need time to digest. If people understood HFT more I do belive it wouldn't be seen as such a nefarious business practice. But since it is relatively difficult to pin down regulating it seems to be the antidote du jour for many mis-diagnosed problems in the financial sector.

    As far as regulating the industry there is a saying in chess where "the threat is greater than the execution", or rather adverse affects from regulating it could prove more costly than the existance of the market itself.

    1. Thank you! Part of what drew me into it originally was the mysteriousness of it all as well. It turns out it's just code like everything else. Code that needs constant attention, testing, care, revision.

      One of the major challenges of automated trading is that things are based so much in these small statistical edges that come and go in various market environments. From a practical perspective, it doesn't make sense for an HFT player to induce market-disrupting behavior.

      The chess metaphor is good. I was actually thinking about that the other day. Price movements across products happen in these discrete steps. In between there's a lot of posturing for position.

    2. Anonymous3:40 PM

      "If people understood HFT more I do belive it wouldn't be seen as such a nefarious business practice."

      I expect the opposite to occur - this isn't a case of someone providing a valuable service to either side of the transaction in exchange for their cut of the fees (which is why the defensive discussion tends toward falsely trying to suggest they are adding value with liquidity since otherwise it's clearly parasitical).

  2. "Important economic statistics are released on regularly scheduled dates and times. Market participants know these well in advance. If a firm conducts trading that is sensitive to an economic number, the firm appropriately removes bids and offers from the market, and possibly closes its position, prior to the number's release. There's very little advantage to be had."

    This can't be true. If the value was non-trivial then nobody would pay reuters for this info. If it's a competitive risk neutral market, then reuters will capture all the profits. If there' s uncertainty or volatility associated with profits then reuter and buyers will split the profits. Say, they split 50-50. Take the revenue reuters makes from selling this infor and multiply by 2. That's going to be exact amount that someone else in the market will be losing. There may also be costs associated with liquidity drying up as information asymmetry becomes more explicit.

    1. Hey Dan! thanks for the reply.

      I agree that there is "some" value in purchasing the information. It certainly allows more mature firms using more complex pricing models to know where they expect the market to be. As far as being able to "pick off" other players, there isn't very much opportunity.

      Liquidity dries up shortly before every number and returns shortly after — even the ones that don't have the ability to be purchased in advance. Traders use services like RanSquawk which does countdowns and announces every number as they come out. It's a pretty interesting thing to watch.

  3. Corkscrew7:28 PM

    This is a broad definition because the field is broad. Loosely, the three main categories of automated trading strategies are (1) latency arbitrage, (2) market making, and (3) statistical arbitrage.

    I think that what bothers people is the fact that, while all three categories use HFT, only the first really needs a low-latency connection. Market making can happen at any speed. So can statistical arbitrage* if (as commonly believed) markets are multifractal.

    Since latency arbitrage is a zero-sum game, this seems highly wasteful. It certainly doesn't lead to a better allocation of scarce economic resources.

    * Unless of course you're relying on spotting patterns faster than anyone else. I'd argue that this is less socially valuable than other forms of statistical arbitrage, as the requirement to make snap decisions means that the patterns you "spot" have a higher chance of being spurious.

    1. Thanks Corkscrew. I agree. Plus I find (2) and (3) to be a lot more fun. You do need some amount of speed to be able to successfully make markets — avoiding adverse selection when informed trading is happening.

  4. HFT could have a stabilizing effect on the market

    HFT want to make a buck like everyone else and they are not going to let a market inefficiency be unanswered. When the market fell 1% intraday twitter bomb scare, HFT were the ones that bid the market back up immediately after it was debunked

    If you want to see how bad an illiquid market is, consider the options market for most stocks. If a call option is quoted a $8 bid and $9 ask, the broker will fill you for no less than $8.8 but if you sell you may only get $8.1. A ripoff

  5. Anonymous3:43 AM

    Although I understand that HFT can cause fluctuations in the real market, the only market HFT provides liquidity to is its' own. HFT only exists because it can. All it seems to be is a bet on price, nobody owns anything in any real sense.

    The consequences of it being banned would be zero except of course for those in it. Whilst I understand that not all HFT literature is not peer reviewed that doesn't make it incorrect.


    1. Hey Bill40, thanks for the comment!

      I think the significant decrease in returns to latency sensitive trades over the past 5 years is a good indicator that even firms that aren't specific to HFT trades are getting smarter about their execution. There has always been a big proportion of market activity that is a simple bet on short term price (noise traders) — granted that proportion does appear to have increased.

      The incorrect research I was referring to specifically comes from Nanex. I discuss several big bold face lies here If their research were published under the peer review process, it would have to be retracted for data fraud.

  6. Anonymous9:47 PM

    What exactly would be wrong with slapping a Tobin Tax on the market to slow it down? Finance is getting out of control (measured by corporate profit, GDP, lobbyists, whatever) and I think it's time to throw a little sand in those gears, starting with killing off HFT.

    In this case, the market seems only able to make very expensive workarounds to combat front running. I think government regulation has an effective role to play here.. except for all those aforementioned lobbyists.

    1. Thanks for the comment! I believe your criticisms of finance (the industry as a whole) are more than reasonable. However, a Tobin Tax (a tax per transaction) wouldn't curb any of that bad behavior. High-frequency firms aren't the same firms that you're targeting — the largest HFT firm is likely KCG, and its operating cash flows are 3 orders of magnitude lower than JP Morgan's.

      The real front running (the illegal kind) and insider trading isn't conducted by HFT. It's done by the actual establishment.

      Further, a tax per transaction also wouldn't do anything to curb speed or level the playing field. I think you'll just see liquidity move to more confusing products or overseas.

      One thing we could look at is how much benefit there really is to "price discovery" at one cent increments. (I'm sure there probably is research in this already)

    2. Anonymous1:05 AM

      "Further, a tax per transaction also wouldn't do anything to curb speed or level the playing field. I think you'll just see liquidity move to more confusing products or overseas."

      Enact a tax in Europe, the US and Japan and then where will the bad activity go? Nobody's going to be trading a significant number of Apple stock in Nigeria or Pakistan. And I think after the disaster five years ago people are going to be a little nervous throwing their pension funds into opaque derivatives.

      Otherwise, I don't see how taxing each trade at a level that is higher than the estimated (short term) profit per trade wouldn't slow down trading, whether it was relatively small HFT firms doing it or the big guys. Done right it probably wouldn't affect market making or "arbitrage" very much, either.

  7. So HFT is not a problem, but you propose to eliminate it using latency floors? Got it....

  8. Hey rjh57! thanks for the response. I can definitely empathize with your confusion. I too have once imagined someone making multiple claims that they didn't actually make.

  9. I guess myself I break HFT down into: 1) the raw order flow information; 2)the pattern recognition software.

    The raw order flow information (bid/ask, order depth) is the true private information asymmetry that allows market makers to collect rents. Payment for order flow, and the practice of allowing some people to see the order book (or economic statistics) ahead of others, is what I have an issue with.

    Trading on patterns in the data, meh. People have been using ever-increasing computing power to wring out seasonal patterns (for example), enforce put-call parity, etc. for a long time. In the 1980s it was really hard to compute implied vol for an option even using Black-Scholes. Now, its instantaneous and I can do it in a VB script in Excel, for 9000 rows, over coffee. Trading on patterns is no different.
    I think this market is competitive and there are literally thousands of firms. With so many firms, the odds of everyone using an identical algorithm, resulting in stability, seem small.

    On the other hand, as I have said before, market making itself is a natural monopoly. There is a difference between having (and protecting) private information, and merely ignoring it. Ignoring it is like not doing enough research on used car, too bad so sad for me. However, market makers that have order flow data (or econ stats) should be *required* to make it available to everyone, not parcel it out based on favors or product placement. The raw data is the part that allows market makers to collect rents, and allows people with access to "take advantage" of those that don't.

    So long as the raw data is available on an egalitarian basis, I really don't see an issue with pattern-based trading. In that respect, HFT is not different than what people have been doing with monthly, daily, intraday data as computers have become more powerful.

    1. Hey dwb. I like how you're thinking about it. Patterns and statistical edges are certainly more fun and highly competitive. That's why I left my definition of HFT so broad. Even something as simple as a candlestick strategy is managed much more quickly and efficiently using an HFT platform. Managing dozens of strategies over hundreds of instruments is obviously impossible for a human to do with any sort of reliability. And it's that type of trade that powers most automated-algo-etc firms.

      Can you expand a little bit on your argument that market making is a 'natural monopoly'? I'm not sure I can get to that same conclusion. Let's take something like the e-mini S&P futures traded on the CME. No one has a natural monopoly market making on there. Certainly there are a few firms that are able to be the counterparty much more often than others, but demand for the highly leveraged product can be overwhelmingly unidirectional such that no single firm could possibly monopolize it.

      Customer order flow is an area that I am undecided about. In short, it allows a broker to jump queue priority by getting to choose whether to fill a customer at the current NBBO price or route it to someone who will. I don't have enough experience with the intricacies of that to make any sort of judgment. However, as I gave in the examples in my post, it's not a risk free trade.

      99%+ of economic numbers are released to the public at the same time. It's a normal routine in the markets.

      Are you talking about raw data for each order/fill across all exchanges? If that data were free then the world would be quite a different place.

    2. "Can you expand a little bit on your argument that market making is a 'natural monopoly'?"
      Market making is a natural monopoly for the same reason that Software, Facebook, and certain technological standards are a natural monopoly: Liquidity, order depth, and the ability to execute trades is valuable in and of itself. Financial exchanges and market makers have a network externality: the more people they know, and trade with the more likely they can fill orders, the less risk they can take on their books by pairing off and diversifying the book. Transaction costs decrease as the liquidity pool increases.

      Think of it like this: The value of Facebook is that everyone you know is on Facebook. It's very hard to get everyone to move off Facebook because you have to get a critical mass of people to move as well.

      Trading works the same way. A market maker is valuable precisely because they have a pre-existing network.

      Once a market maker or exchange establishes a critical mass of liquidity, its extremely hard for startups to establish market share. If you want to incent people to move off a platform and trade with you, you have to... pay them for order flow.

      If you clear your positions through the same entity (like the CME or a bank) there is another benefit: you get the diversification benefit of netting your trades. In some cases, directly, because (for example) CME gives you credits on your margin requirements when you hold positions strongly negatively correlated (for example: peak power and gas). Indirectly, you also get a benefit because your PnL is netted and so the variability in margin is smaller when you have a lot of diversified positions.

      Network externality: value in having all your trades at one firm, more value if a lot of other people also trade with that firm.

      The private information about the order flow is independently valuable (assymentric information). Trading on private information is completely separate and distinct from the network effect.

      I am not suggesting that order flow information should be "free." However, market makers, exchanges, etc should be treated like utilities and should not be allowed to discriminate. e.g. offer order flow info to one group over another, at different prices and different times. The same information package (order book) should be available to everyone, for the same price. If you want to utilize the order data to detect patterns, have at it.

      "Dark Pools" of liquidity (any pools) should not be allowed to use the "network effect" to get a bunch or people trading in that pool, and then also be allowed to utilize HFT to benefit off the truly private order flow information they are collecting. That's like... collecting double rents! One from the network effect and another from private information. I am not a fan of market makers trading for their own book.

      I realize that this practice has been tolerated in the past, to "promote liquidity" because as you say sometimes market makers need to take risk on their own books at times of unbalanced demand. In the old days of the stock market, there were firms who specialized in certain stocks for this very reason. When a big call options order came in, they knew the person placing the trade probably knew something and adjusted their order book accordingly. It was easier to monitor one firm and their practices than 5000. I guess I am ok so long as its kept to a bare minimum, and heavily scrutinized. Perhaps for some highly illiquid products its still needed, but overall I see no need in this day and age.

  10. People often miss an important distinction in these discussions. Data and information are not synonymous. Myself, I think of statistics as being like ore processing: modelling and inference draws useful information out of raw data. In the world of finance we would like to think of everyone having the same access to the same information at the same time. This, of course, is not the case even when they have access to the same data at the same time. But supposing they do all have the data, some people are going to be more efficient at drawing the information out of it than other people.

    Criticisms of HFT often sound to me like complaints by people who can't compete either because they don't have the technical know-how or don't want to learn it. But that's not to say the system can't be made more effective with better regulation, just that we shouldn't make a hasty judgement about the use of computer technology to do more efficiently what people have already been doing for centuries. I'm actually pretty excited to see how far this can go. Maybe I'm overly optimistic, but I think information technologies may prove to be a stabilizing force in finance.