HFTurbo

KO | EN

gitlite — search

#javascript #react #typescript #llm #linux #macos #java #windows #cli #awesome-list #security #pytorch #ios #openai

(★ 11)

Attempt at a low‑latency HFT pipeline using commodity hardware and software optimizations

HFTurbo 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# HFTurbo

My attempt at a complete **high-frequency trading (HFT) pipeline**, from synthetic tick generation to order execution and trade publishing. It’s designed to demonstrate how **networking, clock synchronization, and hardware limits** affect end-to-end latency in distributed systems.

Built using **C++**, **Go**, and **Python**, all services communicate via **ZeroMQ** using PUB/SUB and PUSH/PULL patterns. The stack is fully containerized with **Docker Compose** and can scale under **K8s**. No specialized hardware was used in this demo (e.g., FPGAs, RDMA NICs, etc.), the idea was to explore what I could achieve with commodity hardware and software optimizations.

## Summary of Communication Flow
| Step | Service (From) | Service (To) | Protocol | Purpose |
|------|------|----|-----------|----------|
| 1️⃣ | Synthetic Feeder | Market Data Ingestor | **WebSocket (port 8765)** | Stream synthetic market data |
| 2️⃣ | Market Data Ingestor | Trading Algorithm | **ZeroMQ PUB/SUB (port 5555)** | Broadcast tick data |
| 3️⃣ | Trading Algorithm | Matching Engine | **ZeroMQ PUSH/PULL (port 5557)** | Send generated orders based on algorithm |
| 4️⃣ | Matching Engine | Order Submission / Latency Harness | **ZeroMQ PUB/SUB (port 5558)** | Publish executed trades |
| 5️⃣ | Order Submission | Mock Exchange | **HTTP POST (port 5678)** | Send trade confirmations |
| 6️⃣ | Latency Harness | (Self-contained) | **Local timing** | Measure E2E pipeline latency |

1. Feeder generates synthetic tick data (e.g., price = 100.4202132) every 1ms.
2. Ingestor reads the WebSocket and publishes the tick to all subscribers.
3. Algorithm consumes ticks, generates BUY/SELL orders, and sends them to the matching engine.
4. Matching Engine pairs bids/asks, timestamps the match, and publishes trade results.
5. Order Submission forwards matched trades to a mock exchange via HTTP, and the exchange responds with a confirmation (its just a `200 OK` in this case, for simplicity).
6. Latency Harness subscribes to the port opened by the Order Submission service and measures latency across the pipeline.

## Latency Measurements/Discussion

From this work, I was able to achieve an end-to-end latency of ~10 ms for ~100 trades (ran on my machine, an Asus VivoBook with a Ryzen 7 4700U and 16GB of RAM); from synthetic tick generation to order execution and trade publishing. This was measured using a simple latency harness Python script that utilized the timestamps associated with the orders, and calculated the total time taken for a message to traverse the system.

The main bottlenecks in the system from my perspective/research were:

- **Network latency**: Even on a local network, the time taken for messages to travel between services contributed significantly. Using ZeroMQ helped reduce some of this overhead due to its efficient messaging patterns.

- **Containerization overhead**: In real HFT environments, the network interface card (NIC) bypasses the kernel entirely using technologies like DPDK, RDMA, or Onload, allowing network packets to be DMA’d directly into user space. The NIC can provide hardware time-stamping with nanosecond accuracy, and the trading engine typically runs pinned to a CPU core, reading packets directly—enabling sub-microsecond tick-to-trade latency. **Docker**, or any virtualization layer, breaks this model because it adds network abstraction layers (e.g., bridges, veth pairs, etc...), preventing direct kernel bypass, and cannot expose NIC clock synchronization into containers. This explains why my containerized setup achieved latencies on the order of milliseconds rather than the sub-microsecond latencies seen in production HFT systems.

- **Processing overhead**: Serialization, deserialization, and order book management all contributed minor additional delays, though these seemed to generally be dwarfed by network and containerization effects.

## Potential Improvements

The following improvements that I'm considering to explore in future iterations:

- Investigate the use of more efficient serialization formats (e.g., Protobufs, FlatBuffers) to reduce message size and parsing overhead.

- This implementation assumes a single stock symbol, as it's generally common to maintain one order book per symbol. Expanding to multiple symbols would require more complex order book management and could introduce additional latency, but would make the system more realistic, so it's worth exploring.

- I also want to explore more advanced trading algorithms, such as statistical arbitrage or machine learning-based strategies, to see how they perform in this low-latency environment, right now the algorithm is as trivial as it gets (BUY if current price < last price, SELL if current price > last price).

HFTurbo

HFTurbo 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

HFTurbo

HFTurbo 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README