Round Earth Theory of Evolution

I read this article and noticed the arguments it makes are pretty weak.  It compares belief in evolution to belief in the color blue.  I’d say evolution is more like special relativity.  It’s non-intuitive and not obvious, but with enough effort it can probably be understood, analyzed, measured, observed. Continue reading

follow-up on Flash Boys

In my previous post, I discussed market microstructure.  I hypothesized that limit orders directed to exchanges paying for order flow rather than paying market makers would be executed first.  I couldn’t think of a good reason to route limit orders to IEX.  So, I just ran a preliminary test of the hypothesis by sending 5 orders of 100 shares each to various exchanges, including the InteractiveBrokers Smart Router.  Note that the exchange rebate structure has changed since the publication of Flash Boys.  BYX is now the exchange that pays the most for order flow.  Here are the results.

trades and volume

trades and volume

You can see that I traded on BYX first, as expected.  I paid the seller about 20 cents for the order flow.  A minute later, a larger sale came in and I received about 20 cents on each of BATS, ARCA, DirectEdge, and Island.  The Smart Router had chosen Island.  Then, someone sold 100 shares at 19.16 without knowing he could have made $3 more on IEX.  And a minute and a half later, my IEX order got filled.  To state the obvious, it sucks to have a $19.19 bid go unfilled as someone else sells for $19.16.

Here’s a full view of the bids and asks during this time.  You can see how the BYX trade happened while the asset was still being quoted at its original level.  Even though I paid 40 cents more for those 100 shares, it was a better deal than the second set of trades.  They occurred just as a big sale came in and drove the value down around 5 cents ($5 per 100 shares).  And then the IEX trade didn’t occur until the ask came down to $19.19.

bid, ask, and last trade

bid, ask, and last trade

Another way to see that the IEX execution was bad is to think of it this way:  You want to always be buying on the white line and selling on the blue line.  The BYX trade was good because the white line and blue line didn’t move when we bought on the white line.  The second set of trades were okay because even though the white line dropped as soon as those trades occurred, the blue line still remained above the trade price.  The IEX trade was bad because we bought on the blue line.  Someone had announced intention to sell at $19.19 to the entire market and that’s when IEX sent our order over to match with that seller.

I have another experiment planned, so stay tuned.

Flash Boys

I read Michael Lewis’s Flash Boys in a single sitting at the Cambridge Public Library.  For context, I rarely finish books and it usually takes months.  I read it because I needed to understand the pros and cons of routing trades to IEX and couldn’t find enough detail online.  Flash Boys turned out to surpass my expectations, but left me wondering why IEX doesn’t publish a more concise explanation of how the market works and why trades should be routed to them.  Claiming they have a coil of fiber to introduce a delay that slows down HFT is hardly an explanation.

I should mention that I was positively surprised to see Lewis mention both Nanex and Themis Trading, because they really are the best resources for understanding market microstructure and how HFT works.  I’ve been following these problems with the markets for 4 years now, and never imagined it would get so much public attention.

Lewis identifies a few ways in which HFT takes money from investors.

  1. High speed links between exchanges beat customer orders from one exchange to another.
  2. Varying rebate structures cause brokers to route customer orders to where the broker gets paid the most and the customer is therefore least likely to be filled.
  3. Orders in dark pools can be executed at stale prices because they aren’t displayed as part of NBBO.
  4. Sub-pennying.  Lewis actually fails to explain how sub-pennying hurts traders.  There’s a single reference to sub-pennying in the book, but it says nothing about how it works.  See and

I wish the book had a technical summary of all the phenomena seen, so I’ll try to write one here.

First, some background for non-traders.

There are two basic types of trades: market orders and limit orders.  Market orders are for traders who have strong convictions about the direction a stock is moving.  They are designed to get executed immediately at the best available prices in the market.  They’re also called “market taking” because they take whatever prices are given.  Limit orders are for traders who want to guarantee a particular price or are willing to wait for someone else to trade in the opposite direction.  They’re known as “market making” because they make up the prices you see posted in the market.  A market order is like selling your car to a dealer.  You get instant money, but not as much.   A limit order is like listing the car for sale and waiting for a willing buyer.

Secondly, the order in which trades are executed matters a lot.  Imagine you and a dozen other people are trying to sell something of uncertain and changing value.  If you’re all offering the same price, you don’t want to be the last person to sell.  If that happens, it’s because a buyer came and bought everything, which implies that the thing you sold is probably now worth more.  On the other hand, if you’re the first to sell and everyone else is still offering to sell at the same price, that’s great.  You still have the option of undoing the sale by turning around and buying from another seller at the same price.  Furthermore, imagine a buyer comes and offers one of the sellers a dollar less to buy the thing and the seller agrees.  If you can jump in after the agreement, push the seller aside by offering to save the buyer an additional penny, the buyer probably wouldn’t mind, you’d still be able to undo your trade at a 1 cent loss, and the seller you displaced would feel unfairly treated.

Now, on to the 4 HFT phenomena.

  1. Phenomenon 1 (fastest links between exchanges) only affects market taking orders, not market making orders.  Katsuyama complains that when he sees shares offered on multiple exchanges and sends an order to buy all of them, only some of them get filled.  For years, traders have known that the way to get around this is to release the orders such that they all arrive at nearly the same time.  But maybe Katsuyama discovered it first and it just took a long time for Lewis to write the book.  By the way, you also have to flag orders as ISO (intermarket sweep order) so that the receiving exchange doesn’t wait to confirm that there isn’t a better price at another exchange.  By that time, the available price would have disappeared.  IEX helps a lot here by providing such sweep order functionality to traders who don’t have direct connections to all the exchanges.  I’m surprised nobody mentioned the idea of using synchronized clocks on machines located outside each exchange that release a large order within light meters (tens of nanoseconds) of each other.  I’m sure there are firms doing it.  Maybe Katsuyama didn’t tell Lewis.  But it should be an obvious solution to anyone who thinks about the problem a lot.  Another thought on this phenomenon is that it’s nothing new.  It’s called cross market arbitrage and they used to use telephones.  It’s an unavoidable fact that whoever has the best communication method will keep markets in line with each other.  The unanswered question is, “Why aren’t all customers bringing their trades to the same venue?”  If new exchanges start and pay brokers extra to bring orders there, why don’t all brokers end up directing orders to the single exchange providing the highest kickback “rebate”?
  2. A lot of exchanges provide rebates for market making orders and charge market taking orders.  This incents brokers to direct limit orders to exchanges with the highest positive rebates.  Obviously, those orders will be the last to be filled, since it costs more to take them.  I’m not sure how IEX helps here.  If I post a limit order on IEX that matches the NBBO, it’s not going to get filled before the one on BATS.  Small market orders prefer BATS because they offer a negative rebate (get paid to take markets).  Big market orders often hit BATS first too, as described in the book.  To help limit orders get filled quickly, IEX needs to change the fee structure and pay for market orders while charging limit orders the way BATS does.  And as things are right now, I have no incentive to route small market orders to IEX.  I’m best off sending them to the exchange with the highest negative rebate.  The only counterargument is that the prices on certain exchanges could be stale and no longer available, but here we’re talking about prices that stay stable for multiple seconds at a time and your customer order is the first event causing the price to move.
  3. The dark pool incident in which Rich Gates sells stock on an exchange at a price below the price he’s bidding in the dark pool is interesting.  It wouldn’t happen on a lit exchange because the sell order would automatically be re-routed to your standing best bid.  Maybe the lesson is to not leave limit orders unattended in dark pools?  It’s like being willing to sell a car, but not posting it to craigslist or AutoTrader.  You only tell your local dealer that you’re willing to sell the car and wait until he has a customer willing to pay more, at which point he buys it off you.  Why would you do that?  IEX is a dark pool, but it prevents such occurrences using the coil of fiber that delays dealer trades.  What happens is that while the dealer’s attempt to take your stale limit order is looping around in the fiber, IEX learns about the new prices being offered in the market and then blocks your limit order from executing with the dealer because the order could get a better price now on an “away” exchange.  That’s pretty cool.  But it’s not well explained in the book.  I bet most readers don’t think through exactly how introducing a delay helps at all in protecting against HFT because the fast traders will always be faster than the slow traders.  Lewis should have been very clear that the delay effectively solves the problem of the SIP being too slow and ensures that the exchange knows the NBBO before the trades are executed instead of learning of it too late.  In summary, IEX seems to be a properly functioning dark pool, but I can’t think of any compelling reason to post limit orders in a dark pool instead of publicly at an exchange.  Dark pools seem to be for traders unable to mask their order size by breaking up the order.
  4. Sub-pennying occurs when a market order is about to take a limit order, but then an HFT jumps in front of the limit order by 1/100th of a cent as “price improvement” for the market order.  The market taker saves a penny while the original market maker remains unfilled and protects the HFT from the price immediately worsening.  The end effect is similar to what happens in dark pools (an HFT disrupts two traders from trading with each other).  But sub-pennying happens on public exchanges.  I don’t know a way around it.  It really sucks when you’re at the front of a queue of limit orders and an HFT takes your fill for a meaningless amount of money.  It’s not clear how IEX helps here.  I assume trades executed on IEX only occur at whole cents, so there is no sub-pennying.  But market takers don’t really care so much about sub-pennying because they’re still getting nearly the same price.  And not having your limit orders sub-pennied only helps if people are routing market orders to the exchange where your order is posted.  Therefore, I think IEX should offer negative rebates and pay market takers to bring business to the exchange.

The midpoint peg order offered at IEX seems like an appealing concept.  Bad trade executions are ones where the counterparty immediately knows he got a good deal.  When you take part of a posted market and don’t push the price to the next level, the market makers are happy, which is bad for you.  When you are the last limit order to get filled at a certain price, you have an immediate loss.  The right way to trade is to take the market when you have high conviction about which way the market is going or have limit orders near the top of the book.  If neither of those are possible, having your trades execute halfway between the bid and offer is nice because you know your counterparty is trading out of a need to trade and not to immediately make a profit.  The trouble is that midpoint peg orders can be discovered by probing the market and then traders can manipulate the bids or offers to change the midpoint before executing against your trade.  For instance, say I’m bidding for 10,000 shares and the stock is 24.82 at 24.84.  Someone can send a midpoint buy and a midpoint sell to see which way there’s trading interest to determine that I’m a buyer.  Then, buy all the offers at 24.84, pushing the market to 24.84 at 24.85.  Immediately send a midpoint sell to get rid of the shares just purchased for a half cent profit on each.

There’s actually nothing wrong with the old-fashioned market model of price-time priority and a single exchange.  Your limit orders are treated fairly.  IEX could be that market, except that it’s a dark pool right now.  It also has to deal with a lot of the good customer market order flow being internalized or going off to exchanges that pay negative rebates.

Finally, here are some excelllent posts by Dennis Dick that clearly explain the most egregious flaw in how the stock market currently works.  Until there’s a legislative fix, do the right thing and switch to a broker who allows you to directly route your orders to public exchanges.  We’ll all save money and free up the country’s top technical talent for solving real problems.



Thoughts on the Knightmare

The Knightmare is what we call an event during which Knight Capital, at the time a large publicly traded electronic trading group, lost $460 million in about 45 minutes.  If the error had gone undetected longer, Knight would have gone bankrupt and broken the market by not paying for its trades.  That such a thing could happen illustrates a problem with how our markets work.

When you or I place an order to buy stock, our brokerage firm first checks how much money is in our account and how many other outstanding trade orders we have that could reduce our available funds.  The brokerage firm only sends the order to an exchange after confirming that we have sufficient funds.

We cannot trade directly on the New York Stock Exchange because there would be nobody to guarantee that we can actually pay for the trade.  The stock exchange has members, known as dealers or brokers, who pay expensive membership fees to be able to trade directly.  Brokers are the ones who bring customer orders (such as yours or mine) to be filled.  Dealers are there to take the other side of our order because there may not be another customer who wants the exact opposite trade at the same moment.  Both brokers and dealers have to prove that they have adequate money and risk management controls in place to make their trades without the exchange checking their ability to trade all the time.  Part of this process involves going through a “clearing firm”, usually a very large financial institution, who holds their money and securities.  The clearing firm checks their financial status at the end of every day.  The clearing firm also guarantees that all the trades will be paid for.  To avoid the possibility of a small dealer with a few million dollars buying billions of dollars of stock, the clearing firm will monitor the dealer’s finances throughout the day.  Still, the process is a little backwards, with the trade being executed first, then the ability to pay for it being checked.

Knight Capital had computers sending automated orders to the exchanges with no systems verifying how much money it would cost to fill all of them and comparing it to the available funds of the firm.  It seems a little absurd that they are in charge of regulating themselves.  It seems a little like asking a child to decide how many cookies to eat.  Part of the reason trading works this way is that given the volume of orders submitted to exchanges daily, it is technologically impossible to run each one through a validation process.  But maybe requiring exchanges to clear all orders (verify adequate funds) would do two things.  First, it would slow down the high frequency electronic trading that’s creating an overload of data and risks of events such as the Knightmare.  Second, it would lower the barrier for customers to bypass brokers and trade directly on the exchange.

Tradesports / Intrade and various online currency trading sites already handle both exchange and clearing functions at the same time.  To do so, they developed some amazing computer technology such as the LMAX Disruptor  I’d like to see that approach applied to stock trading.  But it’s hopelessly complex from a legal standpoint.  Ideally, regulators would require all exchanges to clear all their orders.  But that’s unlikely because it’s a huge change and many in the financial industry would lobby against it.  A company could attempt to offer a “fair” exchange with equal access for all retail and professional traders, but it would have to somehow allow itself to be detached from the national market and exempt from Reg. NMS (the national market system).

I think a requirement for all orders to pass through a clearing stage is still a better solution to “high frequency trading” problems than a financial transaction tax.  If the tax can overcome lobbying and aversion to change, a requirement that directly lowers risk and benefits retail traders should be more palatable to regulators and financial market participants.


I was playing a game of Settlers of Catan (or Die Siedler von Catan) with some friends and made an observation about my playing style. I definitely tried to win, but I would avoid placing settlements on opposite corners of hexes or otherwise doing anything to harm the total productivity of the board. I also overinvested in strategies that led to greater long term production at the expense of scoring points in the short run. Another player pointed out that the only objective of the game is to achieve 10 points before any other player. It doesn’t matter if you’re producing more resources or would have been the first to 20 points because the game ends as soon as someone gets 10 points. It doesn’t matter if the board is overpopulated or not once the game ends. You’re more likely to win if you can make the board less inhabitable for your opponents, so wasteful plays can help.

In real life, I am a conservationist because I know that the world lives on once I die. I feel connected to society and want to act in ways that are collectively best for everyone. I believe in policies that work in perpetuity, not 4 years, not my lifetime, or any other end of the world. That’s what sustainability is about.

One way I practice this is at the grocery store I purchase packages that may be slightly disfigured or the milk from the front with the earliest expiration date. If everyone always reached to the back to get the freshest milk, stores would routinely have to discard the older milk that doesn’t sell. If nobody buys the disfigured packages, they get thrown out. There’s nothing good about systematically wasting food.

Which milk do you buy?