Distributed-SIMD-Matrix-Multiplication-Engine

KO | EN

gitlite — search

#javascript #python #hacktoberfest #go #golang #rust #deep-learning #macos #java #cli #kubernetes #vue #security #ios

Distributed-SIMD-Matrix-Multiplication-Engine

(★ 11)

Low-level C++20 SIMD matmul engine behind a Go/gRPC gateway, with SHA-256 request hashing, Redis caching, and a live terminal dashboard.

Distributed-SIMD-Matrix-Multiplication-Engine 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Distributed SIMD Matrix Multiplication Engine

[![CI](https://github.com/yulchanshin/Distributed-SIMD-Matrix-Multiplication-Engine/actions/workflows/ci.yml/badge.svg)](https://github.com/yulchanshin/Distributed-SIMD-Matrix-Multiplication-Engine/actions/workflows/ci.yml)

A polyglot demo of low-level CPU optimization behind a decoupled network
boundary. A bare-metal C++20 engine does the math, hand-tuned for cache locality
and SIMD, while a Go gateway fronts it with validation, a Redis result cache, and
a live terminal dashboard. The two talk over gRPC.

The point isn't "multiply matrices." It's to show, end to end, how the *same*
computation gets roughly 6x faster from memory-layout and vectorization work, and
how a cache plus a clean network boundary turn that into a fast, observable
service.

- **Data plane, C++20 (`cpp-worker/`):** contiguous-memory `Matrix`, cache-aware
  transpose, NEON SIMD matmul. Served over gRPC on `:50051`.
- **Control plane, Go (`go-gateway/`):** `net/http` API, SHA-256 request hashing,
  Redis cache, gRPC client, and a Bubble Tea telemetry TUI. Public on `:8080`.

## Demo
▎ Live terminal dashboard polling the gateway's /stats while 60 concurrent workers hammer the API. Total requests, cache hit/miss ratio, P90 latency, and dependency health all update in real time, with every request streaming into the event log below.
<img width="1300" height="760" alt="demo" src="https://github.com/user-attachments/assets/c6c3c32f-1fe0-49df-b80a-7519ac9f45d3" />

## Architecture

```
                 HTTP/JSON                 gRPC / HTTP-2
   ┌────────┐   :8080     ┌──────────────┐   :50051      ┌──────────────┐
   │ client │────────────▶│  go-gateway  │──────────────▶│  cpp-worker  │
   └────────┘             │  (control)   │◀──────────────│  (data/SIMD) │
        ▲                 └──────┬───────┘   result       └──────────────┘
        │  JSON result           │ :6379
        │                   ┌────▼────┐
        └───────────────────│  redis  │  SHA-256(request) then cached result
              X-Cache: HIT  └─────────┘
```

The gateway validates the JSON (`A.cols == B.rows`), hashes the request, and
checks Redis. A **cache hit** returns the stored JSON directly. A **miss**
forwards to the C++ engine over gRPC, caches the finished response (24h TTL), and
returns it. The gateway does no math; the engine does no I/O beyond gRPC.

## Performance

Apple Silicon (arm64), `-O3 -march=native`. Two scopes that are not comparable,
so they are reported separately.

**Compute engine, in-process** (the `multiply_*` call only, "how fast is the math"):

| Optimization      | 512x512     | 1024x1024    | Speedup vs naive |
| ----------------- | ----------- | ------------ | ---------------- |
| Naive triple-loop | 91.4 ms     | 1028.9 ms    | 1.0x             |
| Transpose         | 42.1 ms     | 425.9 ms     | ~2.4x            |
| SIMD (NEON)       | **19.0 ms** | **172.0 ms** | **~6x**          |

**End-to-end request** (JSON + hash + gRPC + compute + encode, "how fast a client
gets an answer"):

| Stage      | 512x512 | 1024x1024 | Speedup     |
| ---------- | ------- | --------- | ----------- |
| Cache miss | 69.3 ms | 361.7 ms  | 1.0x        |
| Cache hit  | 35.1 ms | 127.5 ms  | up to ~2.8x |

At 1024x1024 the engine's own compute is about 172 ms (exactly the SIMD figure above),
and the other ~190 ms of a miss is serialization: parsing an ~8 MB JSON body,
hashing, gRPC, and re-encoding. A cache hit skips both the compute and the
re-encode, which is why it pays off more as matrices grow.

## Quick start

### Option A: Docker (recommended, nothing to install but Docker)

```bash
# build + run redis, cpp-worker, and go-gateway on one network
docker compose up --build

# in another terminal: send a request (repeat it to see X-Cache flip MISS to HIT)
curl -si localhost:8080/api/v1/compute -d '{
  "matrix_a": {"rows":2,"cols":2,"data":[1,2,3,4]},
  "matrix_b": {"rows":2,"cols":2,"data":[5,6,7,8]}
}' | grep -iE 'x-cache|result'
```

Live dashboard (interactive, starts the stack if needed):

```bash
docker compose run --rm tui
```

### Option B: Local (run each process yourself)

```bash
# 1. Redis (a container is easiest)
docker compose up -d redis

# 2. C++ engine
cd cpp-worker
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build
./build/matrix_server                # listens on :50051

# 3. Go gateway (new terminal)
cd go-gateway
go run .                             # listens on :8080

# 4. Optional: dashboard + load generator (new terminals)
cd go-gateway && go run ./cmd/tui
cd go-gateway && go run ./cmd/stress --workers=50 --duration=60s --hit-ratio=0.7
```

No Redis? Run the gateway with `REDIS_ADDR="" go run .`. Caching is disabled
(fail-open), so every request computes.

## System requirements and what you need

| Path       | You need                                                                              |
| ---------- | ------------------------------------------------------------------------------------- |
| **Docker** | Docker Desktop / Engine + Compose v2. That is all. `cpp-worker` is pinned `linux/arm64`. |
| **Local**  | Go **1.25+**, a C++20 compiler (clang/gcc), CMake **3.14+**, `protobuf` + `grpc`, Redis |

On macOS the local toolchain is:

```bash
brew install go cmake protobuf grpc      # engine + gateway build deps
# Redis via Docker (docker compose up -d redis) or `brew install redis`
```

The SIMD path uses **ARM NEON** (`float32x4`, 4-wide), so it runs natively on
Apple Silicon and ARM Linux. On x86 the engine falls back to the scalar transpose
path (still correct, no NEON); CI runs on a native ARM runner so the real SIMD
path is exercised. Generated protobuf/gRPC stubs are gitignored and regenerated
by the build (`go generate ./...`, CMake, or `./generate.sh`).

## How it works: the optimizations

Each step keeps the math identical (integer test data makes equality exact) and
only changes how the CPU touches memory.

### 1. Flat, contiguous memory (no heap fragmentation)

`std::vector<std::vector<float>>` scatters every row across the heap, causing
pointer chasing and cache misses. Instead, one allocation with manual 2D to 1D
indexing:

```cpp
class Matrix {
    std::size_t rows_, cols_;
    std::vector<float> data_;                 // ONE contiguous block
public:
    float& operator()(std::size_t r, std::size_t c) {
        return data_[r * cols_ + c];          // row-major index math
    }
};
```

### 2. Transpose B (stop thrashing L1)

The naive inner loop reads B down a column (`B(k, j)` strides by `cols` every
step), so almost every access is a cold cache line. Transposing B once (O(N²))
makes both operands read row-sequentially, staying in L1:

```cpp
Matrix B_T = B.transpose();
for (std::size_t k = 0; k < B_T.cols(); ++k)
    sum += A(i, k) * B_T(j, k);      // A[i][k] and B_T[j][k] both contiguous
```

That single change is about 2.4x at 1024x1024 (verified with Cachegrind: the L1
read-miss rate drops sharply).

### 3. SIMD (4 multiply-adds per instruction, NEON)

With both operands contiguous, the dot product vectorizes cleanly: load 4 floats
from each, fuse-multiply-add into a 4-lane accumulator, then a scalar tail for the
remainder:

```cpp
float32x4_t acc = vdupq_n_f32(0.0f);
std::size_t k = 0;
for (; k + 4 <= B_T.cols(); k += 4) {
    float32x4_t a = vld1q_f32(&A(i, k));      // 4 floats
    float32x4_t b = vld1q_f32(&B_T(j, k));
    acc = vmlaq_f32(acc, a, b);               // 4 fused multiply-adds at once
}
// horizontal-sum acc's 4 lanes, then handle leftover columns scalar
```

Result: about 6x over naive at 1024x1024. The path is guarded by
`#if defined(__ARM_NEON)` with a scalar fallback, so non-ARM builds stay correct.

### 4. Smart Redis caching (skip the work and the re-encode)

Identical requests should not recompute. The cache key is the SHA-256 of the
**deterministically-marshaled protobuf** (not raw JSON, whose field order is
unstable), and the cache stores the **finished JSON response body**:

```go
key, _ := cache.Key("matrix:multiply", protoReq)   // sha256 of deterministic proto
if body, found, _ := h.cache.Get(ctx, key); found {
    w.Header().Set("X-Cache", "HIT")
    w.Write(body)            // replay finished bytes: no compute, no re-encode
    return
}
// miss: compute over gRPC, marshal once, cache that body with a 24h TTL
```

Caching the *response body* (not the protobuf) means a hit also skips the
expensive result re-serialization, which for big matrices is a large share of the
cost. Caching is **fail-open**: if Redis is down, requests still compute.

## HTTP API

`POST /api/v1/compute` with a JSON body of two matrices (row-major flat `data`,
`len(data) == rows*cols`). Successful responses carry an `X-Cache: HIT|MISS`
header. `GET /stats` returns the live telemetry snapshot the dashboard reads.

| Status | Condition                                             |
| ------ | ----------------------------------------------------- |
| 200    | success                                               |
| 400    | malformed JSON, trailing data, or failed validation   |
| 405    | method other than POST                                |
| 413    | body exceeds `MAX_REQUEST_BYTES`                      |
| 502    | upstream engine failure (down or crashed)             |
| 504    | upstream call exceeded `REQUEST_TIMEOUT`              |

### Gateway configuration (env vars)

| Var                 | Default           | Meaning                                     |
| ------------------- | ----------------- | ------------------------------------------- |
| `HTTP_ADDR`         | `:8080`           | HTTP listen address                         |
| `CPP_WORKER_ADDR`   | `localhost:50051` | C++ gRPC engine address                     |
| `REQUEST_TIMEOUT`   | `5s`              | Per-request upstream deadline               |
| `MAX_REQUEST_BYTES` | `33554432` (32MiB)| Max inbound request body size               |
| `REDIS_ADDR`        | `localhost:6379`  | Redis address; empty string disables caching|
| `CACHE_TTL`         | `24h`             | How long results stay cached                |

Under Docker Compose these are set to service DNS (`cpp-worker:50051`,
`redis:6379`); unset on the host they fall back to `localhost`.

## Live telemetry dashboard

A [Bubble Tea](https://github.com/charmbracelet/bubbletea) TUI polls `GET /stats`
and renders throughput, cache hit/miss, P90 latency, dependency health, and a
streaming event log in real time. Drive traffic with `cmd/stress` and watch the
panels move; kill the engine mid-run to see health flip to `DOWN`.

```
 ● GATEWAY ONLINE                              PORT 8080   UPTIME 00:12:45
 ┌ TELEMETRY ───────────────┐ ┌ SYSTEM HEALTH ───────────┐
 │ Total    : 14,502        │ │ Redis     : ● CONNECTED   │
 │ Req/Sec  : 342           │ │ C++ Engine: ● CONNECTED   │
 └──────────────────────────┘ └──────────────────────────┘
 ┌ CACHE PERFORMANCE ───────┐ ┌ LATENCY (P90) ───────────┐
 │ Hits   : 9,200 (63%)     │ │ Overall  : 85.0 ms        │
 │ ████████████░░░░░░  63%  │ │ C++ gRPC : 120.0 ms       │
 └──────────────────────────┘ └──────────────────────────┘
 ┌ LIVE EVENT LOG ─────────────────────────────────────────┐
 │ 14:02:13  CACHE MISS  hash c9f4...  1024x1024  172.0ms  │
 │ 14:02:14  CACHE HIT   hash c9f4...  1024x1024    3.0ms  │
 └─────────────────────────────────────────────────────────┘
```

The load generator (`cmd/stress`) drives a tunable hit/miss mix:

```bash
go run ./cmd/stress --workers=100 --duration=30s --dim=512 --hit-ratio=0.7
```

Flags: `--url`, `--workers`, `--duration`, `--hit-ratio`, `--dim` (square NxN),
`--seed` (0 = time-based, fresh misses each run).

## Project structure

```
proto/matrix.proto      the gRPC contract (source of truth)
cpp-worker/             C++20 engine: Matrix + matmul variants, gRPC server, benchmarks
go-gateway/             Go control plane: HTTP API, cache, gRPC client, TUI, load gen
.github/workflows/      CI: build + test + benchmark the engine on every push/PR
docker-compose.yml      redis + cpp-worker + go-gateway (+ profiled tui)
```

Build and test the engine:

```bash
cd cpp-worker && cmake -B build -DCMAKE_BUILD_TYPE=Release && cmake --build build
./build/matrix_tests        # correctness (every variant equals naive)
./build/matrix_benchmark    # 256/512/1024 timings
```

## What could be improved

This is a single-node, single-threaded-per-request demo. Natural next steps:

- **Multithreading the compute.** The outer matmul loop is embarrassingly
  parallel; an OpenMP `#pragma omp parallel for` or a thread pool over row blocks
  would scale close to linearly with cores.
- **Cache blocking / tiling.** Process sub-tiles that fit L2 to cut memory traffic
  further, on top of the transpose.
- **x86 AVX2/AVX-512 path.** Add a parallel guarded intrinsics path alongside NEON
  so it vectorizes on Intel/AMD too (currently scalar fallback there).
- **Protobuf end to end.** Drop JSON at the edge, or stream it, to remove the
  serialization overhead that dominates large-matrix requests.
- **Horizontal scale.** Run multiple `cpp-worker` replicas behind the gateway with
  load balancing; the stateless engine makes this straightforward.
- **Richer observability.** Export the in-memory `/stats` metrics to Prometheus,
  and add multi-arch (`buildx`) images for x86 hosts.
```

Distributed-SIMD-Matrix-Multiplication-Engine

Distributed-SIMD-Matrix-Multiplication-Engine 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Distributed-SIMD-Matrix-Multiplication-Engine

Distributed-SIMD-Matrix-Multiplication-Engine 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README