I am interested in reading about techniques HFT firms use

I was in this area from 2003 to 2013. Can't think of any reading materials, but it all boils down to ultra low latency (think < 100 microseconds within the local network) and high volumes (think > 30,000 transactions per second). The state of the art may have moved since then, but:

1) Network layer: raw TCP sockets with binary packed messages (sometimes structs directly cast to and from raw buffers, keeping network/host byte order in mind); separate message data dictionary to avoid metadata overhead inside messages.

2) Processing: optimised C++ code; pointers; multi-threaded processes that utilise all available cores; minimal locking and contention between threads (queues with spin locks and bulk transfers); threads directly bound to CPU cores (prevent context switching overhead); prevent heap locks by allocating large object pools at startup;

3) Disk I/O: minimise disk I/O in logic-heavy, low latency threads; application level in-memory disk write queues; prefer append mode writes; application level in-memory caches (basically hash maps of reference data loaded at startup)

4) None of the "trendy" programming practices: no OOP design patterns; no dependency injection; functional programming, but without fluent APIs;

5) No popular frameworks: internally developed solutions optimised for our specific use cases, instead of off the shelf libraries or frameworks (e.g. the messaging middleware was in-house)

There is more, but this is what I could think of, off the top of my head.

You can troll through the job postings of most of these companies and learn about their org structures and tech stacks. When I looked almost nothing is public other than older documentaries. Nanex had some neat wiring diagrams and a few videos clips if you google them. Don't get too excited you're not going to learn about some super secret strategy or anything.

This book was good https://www.amazon.ca/Advances-Financial-Machine-Learning-Ma... in that is sort of explains at a high level how you'd build a company to look at something like this (think it was one of the last chapters). You'll need extremely deep/skilled hardware/networking groups, data acquisitions (think packet capture to data) folks, people to clean and maintenance massive the data repos, internal tool teams, quantitative analysts, traders, management/admin, etc. At this level just getting the fiber to each exchange likely costs many many millions. When you go and look through those job postings you can sort of slot these into your mental model of this flow and start to see how it might be put together.

That book is good but if you are new to the topic it will be almost unreadable. At least it was for me. I had to read it 3-4 times over the years and still find new things as my knowledge grows.

I don't know if rentec is a HFT firm or what strategies they use but this was a really good interview with one of the founders https://www.youtube.com/watch?v=QNznD9hMEh0.

That’s a hard question to answer. The question is far too broad.

HFT is a combination of lots of things: low-level computer architecture knowledge, hardware engineering, computer networking, numerical computing, machine learning, super-computing, market simulation, market microstructure knowledge, risk management, financing, psychology/game-theory, etc etc.

Usually no one person is in expert in all of those areas. Usually, a firm is successful because they put together a world class team with all of those skillsets AND they’re willing to invest heavily in the right things.

And even then, it’s so competitive that you can still fail. It’s quite challenging to get everything right. There are far more ways to fail than to succeed.

I’d suggest you pick some sub-area you’re interested and focus heavily on that.

I don't know very much about HFT, and these books are old, so take this with a cow lick of salt, but I've seen these mentioned on a blog I found here on HN (by 'yummyfajitas), and I found them illuminating:

Algorithmic Trading & DMA by Barry Johnson

Trading and Exchanges: Market Microstructure for Practitioners by Larry Harris

You might also be interested in 'yummyfajitas blog:

https://www.chrisstucchio.com/blog/2012/hft_apology.html

If people have up to date replacements for these books, I'd be super interested.

If you keep asking around, people are gunnuh recommend Reminiscences of a Stock Operator, and it's an enjoyable book but keep in mind that it's basically the fictionalized biography of a problem gambler who happened to operate in the stock market, he traded recklessly and died destitute (by suicide - when you reread it, you start to notice how he never mentions enjoying anything, or talks about the people in his life in more than passing). Don't take it too seriously.

The LMax architecture: https://martinfowler.com/articles/lmax.html

Also read up on branchless programming, lock-free data structures and avoiding heap-allocations.

This was a fun read I stumbled upon a while back: https://web.archive.org/web/20210205014443/https://meanderfu...

I went down this road a few years ago. Reading up on algorithmic trading is a great start. HFT often employs normal algorithmic trading techniques optimized for low latency. There are some algorithms that only work when you have a temporal edge. That is the software side. The hardware side is really interesting as well. If a connection to an exchange can be reduced by a few milliseconds it gives a competitive advantage. There is a lot of money spent to be the fastest because for some types of algorithmic trading being fastest almost means free money. Because of how lucrative a minor edge can be you are not likely to find many details on the bleeding edge of HFT.

Beware, the more you read about the practices of HFT firms the more some of it looks like fraud on a grand scale. For example, front running orders is a common practice that goes unpunished for HFT firms but will land you or I in jail.

I got into algorithmic trading and HFT when I was dabbling in bot trading cryptocurrencies. The wild west of crypto bot trading is pretty much gone now but man it was fun while it lasted.

In addition to the algorithmic trading books suggested I found these links helpful: https://www.quantstart.com/articles/Beginners-Guide-to-Quant...

https://www.youtube.com/watch?v=rB5jJuMP84E - just neat

https://www.youtube.com/watch?v=oUVchh37AO8

https://www.youtube.com/watch?v=bo-F61ZuBDg - 2010 flash crash

I found Headland's blog post about quant trading useful [0] and as well as their other blog posts [1].

Jane Street's podcast and blog are insightful too [2]. I read somewhere that Dr. Simons (one of the founds of RenTech) has this book in his office [3].

0. https://blog.headlandstech.com/2017/08/03/quantitative-tradi...

1. https://blog.headlandstech.com/

2. https://blog.janestreet.com/ and https://signalsandthreads.com/

3. The Econometrics of Financial Markets, https://www.amazon.com/gp/product/0691043019/

KDB+ and Q: https://kx.com/academy/ Julia: https://docs.julialang.org/en/v1/ The C Programming Language: https://www.amazon.com/Programming-Language-2nd-Brian-Kernig...

HFT is relative. But here's one thing that is actually used in many firms: https://github.com/Xilinx-CNS/onload (and correspondingly these network cards: https://www.xilinx.com/products/boards-and-kits/alveo/x3.htm...)

The sniper in mahwah blog is a great reading series on Microwave towers and other information you can glean https://sniperinmahwah.wordpress.com/2019/03/26/4-les-moeres...

A good resource I have been reading is from a former HFT firm founder, Christina Qi

Here you go:

Part 1: https://twitter.com/christinaqi/status/1670221886587047941

Part 2: https://twitter.com/christinaqi/status/1670910949921005568

Part 3: https://twitter.com/christinaqi/status/1673621133885849600

Part 4: https://twitter.com/christinaqi/status/1676135720417910784

Part 5: https://twitter.com/christinaqi/status/1677359287172231168

Max Dama used to have a blog. Don't know if it's still up or archived anywhere.

The main aspects are:

On the trading side, HFT is mainly about what's called "market microstructure." These are relatively simple things that are quirks or features of individual markets. Things that most people don't care about, but can make a large difference in HFT profits include when an order comes in, who gets to trade with it first? How are orders routed between different exchanges? What are the popular types of algorithms that people who place large orders use?

On the tech side:

The main is minimizing tick to trade latency. This includes using techniques as basic as colocation at the exchange's (very expensive) data centers and as involved as designing custom ASICs and anything in between like bypassing the normal network stack.

Unfortunately the real state of the art things are all closely held trade secrets, but I'd start reading the exchange data specs to start getting an understanding of the microstructure side, and you can build some of your own experiments to see how quickly you can go from tick to trade. Can you get it sub 1ms? sub 100 us? 10us?

I've learned a lot from Matt Godbolt's blog https://xania.org/ . He's the author of Compiler Explorer, also works at https://aquatic.com/.

Strategy-wise, the fundamental HFT strategy is to detect large orders from banks or hedge funds as they are filled on one exchange, conjecturing that the order may have been split between multiple exchanges, and the using their speed advantage to front-run the order on other exchanges by a few milliseconds. This is what is describe in Michael Lewis' book Flashboys.

Most of these strategies are simple in concept but extremely difficult in execution. Many of them also rely on fine-grained understanding of the exchange's internal mechanisms, things that most traders would not know or be able to figure out.

I enjoyed my time in HFT, it is a fun game to play for a while, but stressful and ultimately unsatisfying as a career trajectory unless you are making serious cash.

This is fascinating, but is focused on the engineering side and not the modeling: https://www.youtube.com/watch?v=b1e4t2k2KJY

While they are out of date and not too technical (and arguably, quite biased), the two books that I've enjoyed the most on this topic are:

1. Dark Pools, by Scott Patterson, and 2. Flash Boys, by Michael Lewis.

You might find some useful recommendations in this recent thread:

https://news.ycombinator.com/item?id=34766728

Check out books Dark Pools and Flash Boys for some stories about HFT...

https://www.amazon.com/Dark-Pools-Scott-Patterson-audiobook/...

https://www.amazon.com/Flash-Boys-Michael-Lewis-audiobook/dp...

For deterministic low latency infrastructure concerns read the docs for red hat’s real time distribution.

https://access.redhat.com/documentation/en-us/red_hat_enterp...

Last I heard from a finance CTO (years ago), the type of companies that would engage in HFT have a direct line and latency SLA with the specific markets they would trade on. My impression is this is not something you can emulate on a small scale in a consumer setup.

The variety of answers here is telling - doesn't seem to be much consensus. Not sure whether that means various people have very different approaches, or just "those who know don't tell, those who tell don't know".

Isn’t that the entire moat of these companies? Very curious what is public

Tech is mostly FPGA order placer interfacing with a C++ market data decision router and a high end Melanox card. Their kit is inserted as physically close to the exchange matching system as possible. One of the big firms famously reversed the box insertion into the rack and trimmed their Ethernet cables down to minimize the physical length signal had to go (probably theatre but that’s the mentality). Some strategies require good low latency inter exchange connectivity so a comprehensive and well organized microwave network is another bit of the stack. Otherwise it’s largely what you would expect. There’s some mythologizing about these companies but a few notes: they’re often highly insular and their code and stacks can be 10 or more years behind the times, but the kernel of execution will be highly current within that stack. A large part of their moat is that mythology, most people would quit and do some other segment of finance like quant trading or stat arb rather than try to squeeze out some profit in HFT. My view is it’s not a winner takes all market though and there’s a lot of room at the table, but it’s a very challenging market overall. I think bringing in a more modern tech stack that cuts the cable back a little more would be an advantage too.

On the techniques, there’s not a ton explicitly written unlike quant and statarb that have almost too much written about them. The essence is very simple though, you basically use techniques you would use in board games or casino games - look at the structure of the game itself and try to find little gaps in the rules that let you insert yourself. For instance if you can see the order being placed into the book before any march is processed you can try to insert yourself into the match engine before anyone else. That’s low latency trading more than HFT though. Likewise price discrepancies between exchanges (microwaves) is low latency.

HFT would typically be something like a market making strategy where you place lots of small orders around where you believe the market is going playing long and shorts. You place bazillions of orders all the time to pick up small variations in a way that on average is better than random trading. This is high frequency trading because the frequency of the trading is high.

Often they get conflated because they complement each other. Being able to do one often implies you can do the other, so why not? They also sometimes help each other synergistically.

Source: I was there (and I’m building my own as a side project with some long time coworkers)

Are you interested in the tech, the trading techniques or both? Do you have any experience with software development or trading in general, or are you starting completely from scratch?

You probably won't find good resources unless you work in a HFT firm where you can gain the inside knowledge.

There are three techniques:

1. Market making 2. Directional trading 3. Arbitrage

The tech is an enabler of these.

Flash Boys by Michael Lewis was a good read, IMHO.

high end network cards with insane packet buffers and short, fast, expensive fiber runs to their hedgies servers.

Has HFT any value for the society in general?