KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #deepseek-harness #typescript #ai-agent #codex #developer-tools #claude-code #windows #mcp #python #desktop-app

SwiftRDMA

★ 53 Open GitHub ↗

SwiftRDMA -- Exposing RDMA NIC Resources for Software-Defined RDMA Scheduling

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# SwiftRDMA

</div>

<h2 align="center">Software-defined scheduling for shared RDMA NICs</h2>

SwiftRDMA is a C/C++ RDMA runtime and scheduling framework for **colocated RDMA workloads** on shared RNICs. It lifts RDMA resource management into software, where a centralized scheduler monitors application SLO pressure and RNIC contention signals, then applies targeted QP-level actions to reduce interference.

SwiftRDMA treats the Queue Pair (QP) as the main software-visible control unit. Instead of leaving QP placement, reuse, receive credits, and QoS behavior fixed after connection setup, SwiftRDMA adapts them online according to workload goals and RNIC resource pressure. The result is cooperative sharing for latency-critical and best-effort workloads on commodity RDMA deployments, without requiring custom RNIC hardware.

## SwiftRDMA vs. SOTA

Existing RDMA frameworks usually fall into two execution models. The **monolithic model** couples application compute and RDMA I/O on the same run-to-completion cores, which minimizes software handoff overhead but can suffer head-of-line blocking. The **separate model** moves RDMA I/O to communication cores, improving reuse and decoupling, but still leaves application workloads, RDMA I/O, and RNIC resources poorly coordinated.

SwiftRDMA's fundamental difference is that it treats RDMA sharing as a **software scheduling problem** rather than a static provisioning or point-isolation problem. The scheduler reasons over workload SLOs, runtime state, and RNIC resource pressure, then applies control-path and data-path actions to steer QP behavior online.

## Architecture Overview

Modern RNICs contain multiple scarce resources that can independently become bottlenecks: UAR pages for doorbells and blue-flame writes, metadata caches for QP/MR/CQ state, processing units for TX/RX/control pipelines, and ports that interact with the network fabric. SwiftRDMA exposes these domains as scheduling targets instead of flattening them into a single bandwidth-share problem.

At runtime, operators and applications provide policies and SLOs. SwiftRDMA gathers application and RNIC signals, decides whether the current RDMA I/O configuration still satisfies those objectives, and invokes control-path or data-path actions through the RDMA I/O runtime.

<h3 align="left">Key Features</h3>

- **Software-defined RDMA scheduling**: translate workload objectives into runtime control over QP behavior and RNIC resource usage.
- **Multi-domain RNIC control**: coordinate UAR placement, RNIC cache pressure, RX processing-unit health, and port bandwidth instead of treating interference as one flat bottleneck.
- **Signal-action control loop**: combine application latency/queueing signals with RNIC counters, RNR events, port usage, receive-queue state, and runtime metadata.
- **QoS-aware threading model**: dedicate IO threads for latency-critical callbacks while offloading best-effort callback work to worker pools when useful.
- **Unified RDMA runtime**: expose `SwiftClient`, `SwiftServer`, and `SwiftConnection` abstractions for building applications over libibverbs and mlx5.
- **Benchmark and application suite**: includes echo, connection stress, KVS, embedding lookup, KVS-stream colocation, QoS, thread-domain, and FlexEMR-related workloads.

## What SwiftRDMA Controls

<div align="center">
  <table border="0" cellspacing="0" cellpadding="0" style="border: none; border-collapse: collapse; width: auto;">
    <tr>
      <td align="left" style="border: none; vertical-align: top; width: 180px;"><b>UAR controller</b></td>
      <td align="left" style="border: none; vertical-align: top;">Balances QPs across RNIC User Access Region pages to reduce doorbell and MMIO contention between IO threads.</td>
    </tr>
    <tr>
      <td align="left" style="border: none; vertical-align: top;"><b>Cache controller</b></td>
      <td align="left" style="border: none; vertical-align: top;">Adjusts QP reuse to trade off RNIC metadata-cache pressure against per-QP queueing delay.</td>
    </tr>
    <tr>
      <td align="left" style="border: none; vertical-align: top;"><b>PU controller</b></td>
      <td align="left" style="border: none; vertical-align: top;">Manages receive credits to avoid Receive-Not-Ready events and RX-pipeline stalls.</td>
    </tr>
    <tr>
      <td align="left" style="border: none; vertical-align: top;"><b>Port controller</b></td>
      <td align="left" style="border: none; vertical-align: top;">Regulates task-level bandwidth with rate limiting and traffic-class assignment.</td>
    </tr>
  </table>
</div>

## Example Use Cases

<div align="center">
  <table border="0" cellspacing="0" cellpadding="0" style="border: none; border-collapse: collapse; width: auto;">
    <tr>
      <td align="left" style="border: none; vertical-align: top; width: 210px;"><b>Latency-critical KVS</b></td>
      <td align="left" style="border: none; vertical-align: top;">Run microsecond-scale key-value traffic while protecting tail latency from colocated RDMA traffic.</td>
    </tr>
    <tr>
      <td align="left" style="border: none; vertical-align: top;"><b>Embedding lookup</b></td>
      <td align="left" style="border: none; vertical-align: top;">Model high-fanout recommendation-system traffic that stresses QP count, RNIC cache, and callback scheduling.</td>
    </tr>
    <tr>
      <td align="left" style="border: none; vertical-align: top;"><b>Bulk transfer and logging</b></td>
      <td align="left" style="border: none; vertical-align: top;">Colocate best-effort stream, logging, garbage-collection, or memory-compaction traffic with latency-sensitive services.</td>
    </tr>
    <tr>
      <td align="left" style="border: none; vertical-align: top;"><b>QoS experiments</b></td>
      <td align="left" style="border: none; vertical-align: top;">Evaluate traffic-class, VLAN PCP, thread-domain, rate-control, and RNIC-aware CPU-placement policies.</td>
    </tr>
  </table>
</div>

See concrete workloads in [`examples`](examples), [`apps`](apps), [`Baseline_benchmark`](Baseline_benchmark), [`QoS_benchmark`](QoS_benchmark), and [`TD_benchmark`](TD_benchmark).

## SwiftRDMA in Action

Applications initialize the runtime, create a client or server, then send through `SwiftConnection` objects. The top-level runtime API is documented in [`SwiftRDMA/README.md`](SwiftRDMA/README.md).

```cpp
#include "swift-rdma.hpp"

int main(int argc, char **argv) {
    SwiftRDMA::initialize(argc, argv);

SwiftRDMA::SwiftClient client("192.168.1.20", 8000);
    SwiftRDMA::SwiftConnection *conn =
        client.swift_connect("192.168.1.10", 8000);

char *buffer = conn->allocate(4096);
    conn->send(buffer, 4096, nullptr, nullptr, nullptr);
}
```

## Performance Highlights

The SwiftRDMA paper evaluates the system on commodity Mellanox RNICs under colocated latency-critical and best-effort workloads.

- **Up to 6.7x lower tail latency** than static QoS/rate-limiter baselines under phased best-effort interference.
- **Up to 14.5% higher throughput** for latency-critical KVS under dynamic colocation.
- **Microsecond-scale controller overheads**, including lightweight port and PU control paths and optimized UAR/cache signal-action paths.
- **Broad colocation support**, maintaining microsecond-level tail latency across KVS, storage, RPC, embedding, GC, and logging-style workloads.
- **QoS-aware threading gains**, reducing KVS latency slowdown by 30.8% to 88.8% under best-effort embedding callback pressure.

## Installation

### Prerequisites

SwiftRDMA is intended for Linux machines with RDMA-capable Mellanox/NVIDIA RNICs and the libibverbs/mlx5 stack installed.

Typical dependencies include:

- C++17 compiler, `make`, and `clang-format`
- RDMA libraries: `libibverbs`, `librdmacm`, `mlx5`
- Runtime libraries: `glog`, `gflags`, `tbb`, `numa`, `boost_system`, `boost_thread`, `double-conversion`
- Folly and fmt installed in the paths expected by the Makefiles: `/opt/folly` and `/usr/local/lib`
- Optional RNIC telemetry tooling for scheduler experiments, including NeoHost/DOCA-style counters where available

On Ubuntu, the base development packages can be installed with:

```bash
sudo apt update
sudo apt install -y \
  build-essential clang-format \
  libibverbs-dev librdmacm-dev \
  libgoogle-glog-dev libgflags-dev \
  libtbb-dev numactl libnuma-dev \
  libboost-system-dev libboost-thread-dev \
  libdouble-conversion-dev
```

> [!NOTE]
> The checked-in Makefiles assume local Folly and fmt installations. Adjust `CFLAGS` and `LDFLAGS` if your machine uses different install prefixes.

### Build the Runtime

```bash
cd SwiftRDMA
make -j
```

This builds `libswiftrdma.a` and `libswiftrdma.so`, then copies public headers to [`include`](include) and libraries to [`libs`](libs).

### Build the Basic Example

```bash
cd examples
make -j
```

Run the example server and client on RDMA-capable hosts:

```bash
# server
./swift-server --dev=mlx5_1 --gid=3 --iothread_num=1

# client
./swift-client --dev=mlx5_1 --gid=3 --iothread_num=1
```

## Benchmarks

### Echo Parallel Benchmark

The callback-driven echo benchmark is a good first throughput and latency sanity check.

```bash
cd examples/echo-parallel-bench
make -j

# server
./echo-parallel-bench-server \
  --bind_ip=192.168.1.10 \
  --port=8000 \
  --dev=mlx5_1 \
  --gid=3 \
  --iothread_num=8

# client
./echo-parallel-bench-client \
  --client_ip=192.168.1.20 \
  --server_ip=192.168.1.10 \
  --port=8000 \
  --dev=mlx5_1 \
  --gid=3 \
  --iothread_num=8 \
  --num_qps=64 \
  --window_per_qp=32 \
  --payload_size=64 \
  --exec_time=10 \
  --n=true
```

For high-RPS runs, add `--histogram_metrics=true` to avoid expensive end-of-run sorting.

### KVS and Stream Colocation

The KVS-stream benchmark exercises a latency-critical small-message workload colocated with best-effort bandwidth traffic.

```bash
cd examples/kvs-stream-colo-bench
make -j

# server
./kvs-stream-colo-bench-server \
  --bind_ip=192.168.100.1 \
  --kvs_port=8000 \
  --stream_port=8001 \
  --dev=mlx5_1 \
  --gid=5 \
  --kvs_iothread_num=10 \
  --stream_iothread_num=1 \
  --contention=2

# client
./kvs-stream-colo-bench-client \
  --client_ip=192.168.100.1 \
  --server_ip=192.168.100.1 \
  --kvs_port=8000 \
  --stream_port=8001 \
  --dev=mlx5_1 \
  --gid=5 \
  --kvs_iothread_num=10 \
  --stream_iothread_num=1 \
  --kvs_num_qps=20 \
  --stream_num_qps=2 \
  --kvs_window_per_qp=1 \
  --stream_window_per_qp=64 \
  --kvs_payload_size=16 \
  --stream_payload_size=4096 \
  --exec_time=10 \
  --stream_warmup_sec=2 \
  --contention=2 \
  --n=true
```

For VLAN PCP based QoS experiments, see [`examples/echo-pcp-qos-bench`](examples/echo-pcp-qos-bench) and [`examples/kvs-stream-colo-bench/README.md`](examples/kvs-stream-colo-bench/README.md).

### Application Workloads

- [`apps/sr-kvs`](apps/sr-kvs): SwiftRDMA-based key-value store.
- [`apps/embedding-lookup`](apps/embedding-lookup): configurable RDMA embedding lookup benchmark.
- [`apps/sr-emb`](apps/sr-emb): best-effort embedding table workload.
- [`apps/kvs-emb`](apps/kvs-emb): combined KVS and embedding workload.
- [`FlexEMR`](FlexEMR): FlexEMR-related RDMA application components.

## Runtime Configuration

Common runtime parameters include:

| Parameter | Description | Default |
|-----------|-------------|---------|
| `--dev` | RDMA device name used during connection setup. | `mlx5_0` |
| `--gid` | Local GID index. | `3` |
| `--region_size` | Send/receive memory region size per connection. | `1000000KB` |
| `--iothread_num` | Number of IO threads for a `SwiftNode`. | `1` |
| `--contention` | Thread-domain placement mode: `0` maximum contention, `1` thread-domain agnostic, `2` minimum contention. | `1` |
| `--numa_node` | Optional NUMA override for RNIC-aware CPU placement. | `-1` |
| `--num_worker_threads` | Worker threads for offloaded best-effort callbacks. | workload-specific |

SwiftRDMA also includes an RNIC-aware CPU allocator that prefers CPUs local to the selected RNIC, respects process affinity, avoids busy CPUs when possible, and coordinates CPU reservations across cooperating SwiftRDMA processes. See [`rnic-aware-cpu-allocator.md`](rnic-aware-cpu-allocator.md).

## Project Layout

```text
SwiftRDMA/                 Core RDMA runtime and public API implementation
include/                   Exported headers copied by the runtime build
libs/                      Built static and shared SwiftRDMA libraries
src/scheduler/             Scheduler abstraction and RDMA scheduler
src/signal-gather/         Job and RNIC signal collection utilities
src/actions/               Scheduler action hooks
src/policies/              Scheduling policy hooks
examples/                  Echo, QoS, connection, and colocation benchmarks
apps/                      KVS, embedding, and mixed application workloads
Baseline_benchmark/        Baseline benchmark implementations
QoS_benchmark/             QoS benchmark implementations
TD_benchmark/              Thread-domain benchmark implementations
```

## Contributing

Contributions and experiments are welcome. A good development loop is:

```bash
cd SwiftRDMA
make -j

cd ../examples/echo-parallel-bench
make -j
```

Please keep benchmark documentation close to the workload directory, and update this top-level README when adding a new major runtime feature, controller, or application suite.

## Citation

If you find SwiftRDMA useful, please cite the APNet 2025 vision paper:

```bibtex
@inproceedings{huang2025exposing,
  title     = {Exposing RDMA NIC Resources for Software-Defined Scheduling},
  author    = {Huang, Yibo and Qiu, Yiming and Xiao, Yunming and Bhatnagar, Archit and Ratnasamy, Sylvia and Chen, Ang},
  booktitle = {Proceedings of the 9th Asia-Pacific Workshop on Networking (APNET 2025)},
  pages     = {1--8},
  year      = {2025},
  doi       = {10.1145/3735358.3735359}
}
```

Paper: [PDF](https://huangyibo.github.io/files/apnet2025-SwiftRDMA.pdf) | [DOI](https://doi.org/10.1145/3735358.3735359)

SwiftRDMA

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README