HFT-LimitOrderBook

KO | EN

gitlite — search

#hacktoberfest #llm #go #golang #android #machine-learning #java #windows #vue #pytorch #security #ios #openai #open-source

HFT-LimitOrderBook

(★ 17)

Real-time DBN market-data pipeline with a nanosecond-level limit order book, lock-free SPSC queues, and high-speed ClickHouse ingestion. Includes a TCP streamer, Databento DBN decoder thread, LOB engine, and batch writer. Designed for quant dev / HFT interview performance.

HFT-LimitOrderBook 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# HFT Limit Order Book Pipeline  
High-Performance Market-Data Decoder • Lock-Free Concurrency • ClickHouse Ingestion • C++20

This project implements a complete high-performance market-data pipeline that processes Databento DBN MBO data, maintains a real-time limit order book, and streams normalized rows into ClickHouse for downstream analytics.

---

## Overview

The system consists of four parallel components:

1. **DataStreamer**  
   Streams DBN files over TCP at high speed using epoll, mmap, and an optional token-bucket rate limiter.

2. **Decoder Thread**  
   Uses the Databento C++ SDK to decode MBO messages and pushes them into a lock-free SPSC queue.

3. **Limit Order Book (LOB) Thread**  
   Maintains the bid/ask books, applies add/cancel/replace events, and periodically emits JSON snapshots.

4. **ClickHouse Writer Thread**  
   Batches updates in chunks of 5000 rows and inserts into ClickHouse asynchronously.

Each component runs on an isolated thread and communicates via lock-free ring buffers for maximum throughput.

---

## Architecture

```
   DataStreamer (TCP)
           │
           ▼
  Decoder Thread (DBN)
           │
    SPSCQueue<LOBEvent>
           │
           ▼
  LOB Thread (Order Book)
           │
           ├───────► JSONL Snapshots
           ▼
 SPSCQueue<DBRow> (batched)
           │
           ▼
ClickHouse Writer Thread
```

---

## Features

### Market-Data Decoding
- Databento DBN decoding via official SDK  
- Custom TcpReadable class (zero-copy, handles socket edge cases)  
- TCP_NODELAY and TCP_QUICKACK enabled  
- Exception-safe and backpressure-aware

### Lock-Free Message Passing
- Custom SPSC ring buffer (cache-line aligned)  
- No locks in the hot path  
- Low latency and predictable performance

### Real-Time Limit Order Book
- Tick-accurate add/cancel/replace logic  
- Bid/ask stored in sorted `std::map`  
- Each price level tracks quantity and order count  
- Snapshot generation in JSONL

### ClickHouse Integration
- Batches writes for reduced overhead  
- Stores event/receive timestamps, instrument ID, price, size, side, action, sequence  
- Suitable for large-scale analytics, backtesting, and microstructure research

### High-Performance DataStreamer
- Uses mmap to avoid syscalls  
- Epoll for efficient socket readiness  
- Optional rate-limiting with high-precision token bucket  
- Capable of >1.5 GB/s streaming depending on hardware

---

## Current Capabilities

### Snapshot Generation
The order book emits a continuous feed of JSON snapshots throughout the entire duration of the MBO stream.  
The snapshot frequency is configurable and designed to provide enough data points for validation and accuracy checks.

### Order Book Reconstruction
The order book logic maintains:
- correct price-time priority
- clean separation between bid and ask maps
- accurate handling of add, cancel, and replace events
- consistent management of price-level aggregates
- removal of empty levels and prevention of zero-quantity entries

This ensures a full and stable reconstruction over the entire feed.

### Verification and Validation
A dedicated Python validation tool is included.  
It verifies:
- bid prices sorted from highest to lowest
- ask prices sorted from lowest to highest
- correct quantities and order counts
- absence of crossed markets
- valid timestamps
- proper JSON structure
- integrity of all price levels

The validator is suitable for batch testing and CI automation.

### Latency Monitoring
A latency tracker records the processing time of order book updates.  
It reports:
- average latency
- p50, p90, p99, p99.9 percentiles
- minimum and maximum values

The typical p99 latency is in the microsecond range, comfortably meeting the required target of less than 50 milliseconds.

---

## Project Structure

```

src/
├── data_streamer.cpp
├── decoder_thread.h
├── lob_thread.h
├── clickhouse_writer.h
├── TcpReadable.h
├── lob.h
├── pipeline.h
├── pipeline.cpp
├── spsc_queue.h
├── pipeline_events.h
└── main.cpp

````

---

## Build Instructions

### Requirements
- C++20 compiler  
- CMake ≥ 3.27  
- Databento C++ SDK installed  
- ClickHouse C++ client installed  
- Libraries: zstd, lz4, OpenSSL, Threads

### Build

```bash
mkdir build
cd build
cmake ..
make -j$(nproc)
````

Executables produced:

```
DataStreamer
hft_lob
```

---

## Running the Pipeline

### 1. Start ClickHouse

```bash
docker run -d --name clickhouse \
  -p 9000:9000 -p 8123:8123 \
  --ulimit nofile=262144:262144 \
  clickhouse/clickhouse-server:latest
```

Create the ingestion table:

```sql
CREATE TABLE mbo_records (
    ts_event       DateTime64(9),
    ts_recv        DateTime64(9),
    instrument_id  UInt32,
    order_id       UInt64,
    price          Int64,
    size           UInt32,
    side           UInt8,
    action         UInt8,
    sequence       UInt32,
    channel_id     UInt8,
    flags          UInt16
) ENGINE = MergeTree()
ORDER BY ts_event;
```

---

### 2. Start the DataStreamer

```bash
./DataStreamer 5000 path/to/file.dbn
```

### 3. Start the LOB Pipeline

```bash
./hft_lob 127.0.0.1 5000 snapshots.jsonl
```

Snapshots appear in:

```
snapshots.jsonl
```

Data is written to ClickHouse in batches of 5000 rows.

---

## Documentation (Doxygen)

Generate documentation:

```bash
cd build
make doc
```

HTML documentation appears at:

```
build/docs/html/index.html
```

If LaTeX support is installed:

```bash
cd build/docs/latex
make
```

This produces a full PDF containing all modules and class diagrams:

```
refman.pdf
```

---

## Performance

| Component            | Achieved Throughput          |
| -------------------- | ---------------------------- |
| TCP streaming        | **~90–100 MB/s** (~1M msg/s) |
| Decoder → LOB queue  | ~3–7 ns/msg                  |
| LOB processing       | ~80–130 ns/msg               |
| ClickHouse ingestion | 250k–400k rows/sec           |

The system comfortably supports real-time MBO ingestion even under high message rates.

---

## License

MIT License.

HFT-LimitOrderBook

HFT-LimitOrderBook 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

HFT-LimitOrderBook

HFT-LimitOrderBook 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README