Relax

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

★ 571

An Asynchronous Reinforcement Learning Engine for Omni-Modal Post-Training at Scale

다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## Relax: An Asynchronous Reinforcement Learning Engine for Omni-Modal Post-Training at Scale

**Towards Async, Omni-Modal RL at Scale, Just Relax.**

<p>
  <a href="./README.md">📖 English</a> | <a href="./README_zh.md">📖 中文</a>
</p>
</div>

______________________________________________________________________

**Relax** (**R**einforcement **E**ngine **L**everaging **A**gentic **X**-modality) is a high-performance reinforcement learning post-training framework open-sourced by the Xiaohongshu AI Infra Team for multimodal large language models. Built on Ray Serve with a service-oriented architecture, Relax uses Megatron-LM as the training backend and SGLang as the inference engine. Through the [TransferQueue](https://github.com/redai-infra/TransferQueue) data transfer system, it achieves complete decoupling of training and inference, supporting end-to-end multimodal RL training from text to images, videos, and audio.

______________________________________________________________________

## ✨ Highlights

- 🌐 **Full Omni-Modal Training** — One unified framework for text, vision, and audio RL — one of the few systems capable of end-to-end Omni model (Qwen3-Omni) post-training
- ⚙️ **Service-Oriented Six-Layer Architecture** — Every role is an independent Ray Serve deployment, with native service-level elastic scheduling and fault recovery
- ⚡ **Fully Async via TransferQueue** — Rollout, Actor, ActorFwd, Reference, and Advantages run on independent GPU clusters with streaming data exchange and configurable staleness
- 🔁 **Hybrid Mode** — Separate Actor/Rollout placement groups with TransferQueue streaming, while ref / actor_fwd / advantages run in-process on the actor — pairs `--balance-data` with sub-batched forward to minimize GPU waste
- 🤖 **Agentic RL** — Multi-turn interaction, loss masking, flexible termination, and VLM multimodal context carry-over for closed-loop "execute → observe → decide" training
- 🔀 **Elastic Rollout Scaling** — Dynamically grow/shrink inference engines mid-training via HTTP REST API, with same-cluster (`ray_native`) and cross-cluster (`external`) federation modes
- 🧠 **Rich Algorithm Suite** — PPO, GRPO, GSPO, SAPO, CISPO, and On-Policy Distillation out of the box, with pluggable rewards and built-in **GenRM** (LLM-as-judge) mode
- 🚀 **Megatron + SGLang Backends** — Megatron-LM (TP/PP/CP/EP) for MoE and deep models, SGLang for high-throughput inference, DCS for NCCL-broadcast weight sync
- 📦 **Production-Ready Ops** — HealthManager auto-recovery, centralized Metrics Service (WandB / TensorBoard / ClearML), and Apprise real-time notifications

______________________________________________________________________

## 📢 News

| 📣 Updates                                                                                                                                                                                         |
| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **\[05/26/2026\]** 🔁 New **Hybrid** execution mode — streaming data + in-process ref/actor_fwd, with `--balance-data` support. See the [Hybrid Training Guide](docs/en/guide/hybrid-training.md). |
| **\[05/11/2026\]** 🚀 Support for Qwen3.6 series models (text + VLM)!                                                                                                                              |
| **\[04/15/2026\]** 🎉 Relax is now open-source!                                                                                                                                                    |

______________________________________________________________________

## 🏗️ Architecture

Relax adopts a **six-layer service-oriented architecture** where every role is deployed as an independent [Ray Serve](https://docs.ray.io/en/latest/serve/index.html) deployment, cleanly separating orchestration, components, engines, backends, and distributed capabilities:

| Layer             | Responsibility                                                                                                               |
| :---------------- | :--------------------------------------------------------------------------------------------------------------------------- |
| **Entrypoints**   | `train.py` — signal handling, CLI parsing, Ray cluster connection, Controller launch                                         |
| **Orchestration** | `Controller` (training loop, global restart), `Service` (placement groups, lifecycle), `Registry` (role & algorithm mapping) |
| **Components**    | Ray Serve deployments: **Actor**, **Rollout**, **Critic**, **ActorFwd**, **Advantages**, **GenRM**                           |
| **Engine**        | SGLang rollout engine, pluggable reward functions, request router, data filters                                              |
| **Backends**      | **Megatron-LM** training backend (TP/PP/CP/EP) and **SGLang** inference engine                                               |
| **Distributed**   | Ray Actor groups (RolloutManager / GenRMManager) and **DCS** (Distributed Checkpoint Service) for NCCL/GLOO weight sync      |

**Three execution modes** are supported:

- **Colocate (Sync)** — Actor and Rollout time-share the same GPUs; Rollout writes a full batch to TransferQueue, then yields GPUs for training. Memory-efficient for constrained hardware and strict on-policy (`max_staleness=0`).
- **Fully Async** — Actor, Rollout, ActorFwd, Reference, and Advantages run on **independent GPU clusters** in parallel, exchanging data through TransferQueue and syncing weights asynchronously through DCS for maximum throughput with configurable staleness.
- **Hybrid** — Actor and Rollout sit on **separate GPU placement groups** (like Fully Async) and exchange data via TransferQueue with configurable staleness, but ref / actor_fwd / advantages run **in-process on the actor's own GPUs** via `TensorBackuper` + `_switch_model` (like Colocate). Enables streaming pipelines plus `--balance-data` without paying for standalone ref/actor_fwd services.

> 📖 Learn more: [Architecture Guide](docs/en/guide/architecture.md) · [Fully Async Training](docs/en/guide/fully-async-training.md) · [Hybrid Training](docs/en/guide/hybrid-training.md) · [Elastic Rollout Scaling](docs/en/guide/elastic-rollout.md)

______________________________________________________________________

## 🧠 Supported Algorithms

> 📖 Adding a new algorithm is straightforward — implement a service class, register it in the `ALGOS` registry, and you're done.

______________________________________________________________________

## 🤖 Supported Models

Relax is designed for **omni-modal RL training** — text, vision, and audio in one unified framework. Multimodal data is configured via the `--multimodal-keys` flag, with complete image/video/audio processing pipelines under `relax/utils/multimodal/` for fine-grained control over image token counts, video frame sampling, and audio sample rates.

> 📖 New architectures are integrated via [Megatron Bridge](relax/backends/megatron/mbridge/) for automatic HF ↔ Megatron weight conversion.

______________________________________________________________________

## 📦 Installation

The recommended way to run Relax is via the official Docker image, which ships with all CUDA, PyTorch, Megatron-LM, SGLang, and Ray dependencies pre-installed and version-matched.

```bash
# Pull the official image
docker pull ghcr.io/redai-infra/relaxrl:latest

# Launch a container with GPUs, shared memory, and your workspace mounted
docker run -it --gpus all --ipc=host --network=host \
  -v /path/to/your/workspace:/root \
  ghcr.io/redai-infra/relaxrl:latest bash

# Inside the container
git clone https://github.com/redai-infra/Relax.git /root/Relax
cd /root/Relax && pip install -e .
```

> 📖 For GPU driver requirements, multi-node setup, and persistent storage mounts, see the [Installation Guide](docs/en/guide/installation.md).

______________________________________________________________________

## 🚀 Quick Start

Three end-to-end tasks cover **text**, **vision-language**, and **omni-modal** training. Each task downloads a public HuggingFace dataset and model, then launches training with a single script. Set `EXP_DIR=/root` (or wherever your models and datasets live) and the scripts will locate them automatically.

### Task 1 — DAPO Math (Text, 8 GPUs)

Train Qwen3-4B on [`dapo-math-17k`](https://huggingface.co/datasets/zhuzilin/dapo-math-17k) with GRPO. Reward is rule-based answer extraction plus symbolic math verification.

```bash
hf download --repo-type dataset zhuzilin/dapo-math-17k --local-dir /root/dapo-math-17k
hf download Qwen/Qwen3-4B --local-dir /root/Qwen3-4B

cd /root/Relax && export EXP_DIR=/root
bash scripts/training/text/run-qwen3-4B-8xgpu.sh
```

### Task 2 — Open-R1 (Vision-Language, 8 GPUs)

Train Qwen3-VL-4B on [`multimodal-open-r1-8k-verified`](https://huggingface.co/datasets/lmms-lab/multimodal-open-r1-8k-verified) with GRPO using the `openr1mm` reward.

```bash
hf download --repo-type dataset lmms-lab/multimodal-open-r1-8k-verified \
  --local-dir /root/multimodal-open-r1-8k-verified
hf download Qwen/Qwen3-VL-4B-Instruct --local-dir /root/Qwen3-VL-4B-Instruct

cd /root/Relax && export EXP_DIR=/root
bash scripts/training/multimodal/run-qwen3-vl-4B-8xgpu.sh
```

### Task 3 — AVQA (Omni-Modal: Image + Audio, 16 GPUs / 2 nodes)

Train Qwen3-Omni-30B-A3B on [`AVQA-R1-6K`](https://huggingface.co/datasets/harryhsing/AVQA-R1-6K) with GRPO and a multiple-choice reward.

```bash
hf download --repo-type dataset harryhsing/AVQA-R1-6K --local-dir /root/AVQA-R1-6K
hf download Qwen/Qwen3-Omni-30B-A3B-Instruct --local-dir /root/Qwen3-Omni-30B-A3B-Instruct

cd /root/Relax && export EXP_DIR=/root
bash -x scripts/entrypoint/spmd-multinode.sh \
  scripts/training/multimodal/run-qwen3-30B-A3B-omni-16xgpu.sh
```

Once running, you should see logs like:

```text
Finish rollout 0/200
training step 0/200
```

Checkpoints are saved in Megatron DCP format; convert them to HuggingFace weights with `scripts/tools/convert_torch_dist_to_hf_bridge.py`.

> 📖 Full walkthrough: [Quick Start Guide](docs/en/guide/quick-start.md) · [Customize Training](docs/en/guide/customize-training.md) · [Configuration Guide](docs/en/guide/configuration.md)

______________________________________________________________________

## ⚡ Key Features

### Fully Async Training via TransferQueue

In fully-async mode, Rollout, Actor, ActorFwd, Reference, and Advantages run on **independent GPU clusters** in parallel. Three mechanisms make this efficient:

- **StreamingDataLoader** — Actor begins consuming samples as Rollout incrementally writes them to TransferQueue, eliminating GPU idle time between phases.
- **Configurable staleness** — `--max-staleness` precisely controls how off-policy training data can drift, flexibly balancing on-policy accuracy and throughput.
- **DCS weight sync** — After each training step, weights are NCCL-broadcast from Actor to Rollout/ActorFwd/Reference via the Distributed Checkpoint Service, overlapped with the next training computation.

### Agentic RL

Relax provides first-class support for multi-turn, closed-loop "execute → observe → decide" training:

- **Multi-turn sampling with loss masking** — model outputs (mask=1) are cleanly separated from environment observations (mask=0) so only model actions participate in training.
- **Environment / Rollout decoupling** — a standard `BaseInteractionEnv` interface (`reset`, `step`, `format_observation`) lets environments evolve independently of the sampler.
- **VLM multimodal context carry-over** — `image_data` on the Rollout side and `multimodal_train_inputs` on the training side are incrementally merged each turn so visual observations concatenate correctly.
- **Flexible termination** — combine `max_turns`, token-budget exhaustion, and env-signalled `done`. The DeepEyes example demonstrates Agentic multi-turn GRPO with Qwen3-VL-30B-A3B.

### Elastic Rollout Scaling

Since 60–70% of RL training time is spent in the Rollout phase, Relax exposes **HTTP REST APIs** to dynamically add or remove inference engines mid-training without interrupting the training loop:

- **`ray_native`** mode — specify a target engine count; Relax allocates resources and launches new SGLang engines inside the current Ray cluster.
- **`external`** mode — register SGLang engines already deployed in other clusters for cross-cluster federated inference on preemptible or idle resources.

Scaling is asynchronous, idempotent, mutually exclusive, and supports graceful drain-and-remove plus cancellation with rollback. Engines from startup parameters are protected; only dynamically added engines can be scaled in.

### Megatron Training Backend & SGLang Inference

Training uses **Megatron-LM** with full Tensor / Pipeline / Context / Expert parallelism for MoE and ultra-deep models. Inference uses **SGLang** with process-lifecycle management. New model architectures plug in through Megatron Bridge for automatic HF ↔ Megatron weight conversion.

### Pluggable Reward Hub

Built-in rewards for math (DeepScaler, DAPO), GPQA, F1, IFBench, multiple-choice, multimodal Open-R1, and **GenRM** (generative LLM-as-judge). Add a custom reward by dropping a single file into `relax/engine/rewards/`.

### Production Operations

- **HealthManager** — heartbeat monitoring with two-tier auto-recovery (in-place restart first, global restart as fallback).
- **Metrics Service** — centralized Ray Serve deployment that fans out to TensorBoard, WandB, and ClearML.
- **Notifications** — real-time training alerts via Apprise (Slack, WeChat, email, and more).

______________________________________________________________________

## 📚 Documentation

Full bilingual documentation is available at **[redai-infra.github.io/Relax](https://redai-infra.github.io/Relax)**.

______________________________________________________________________

## 🧪 Examples

______________________________________________________________________

## 🧩 Projects Built upon Relax

| Project                                                  | Description                                                                                                                                                             |
| :------------------------------------------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [HyperEyes](https://github.com/DeepExperience/HyperEyes) | A parallel multimodal search agent that uses Relax for efficient RL training, combining visual grounding and retrieval to search across multiple entities concurrently. |

______________________________________________________________________

## 🤝 Contributing

We welcome contributions of all kinds! Please read our [Contributing Guide](docs/en/guide/how-to-contribute.md) to get started.

______________________________________________________________________

## 🛠️ AI Coding Skills

Relax ships a set of [Claude Code](https://claude.ai/code) slash-command skills under `skills/` to accelerate development and operations. Invoke them in Claude Code with `/skill-name`.

| Skill                | Description                                                                                     |
| :------------------- | :---------------------------------------------------------------------------------------------- |
| `/code-review`       | Expert review of git changes — SOLID violations, security risks, ML/distributed training issues |
| `/debug-hang`        | Automatically diagnose Ray distributed training hangs — collects call stacks and actor states   |
| `/dev`               | Develop and debug Relax code; submit and monitor jobs on a remote Ray cluster                   |
| `/doc-writer`        | Write and maintain bilingual (English + Chinese) VitePress documentation                        |
| `/git-commit`        | Create Conventional Commits with rich markdown body and auto-run pre-commit hooks               |
| `/model-integration` | Step-by-step guide for integrating new model architectures into the training pipeline           |
| `/perf-doctor`       | Audit training launch scripts for performance and GPU memory misconfiguration                   |
| `/ssh-ray-cluster`   | SSH into a remote Ray cluster head node to inspect status, logs, and debug jobs                 |
| `/verl-to-relax`     | Migrate RL recipes from verl to Relax (rewards, tool envs, launch scripts)                      |
| `/creating-skills`   | Guide for authoring new Claude Code skills following Anthropic best practices                   |

______________________________________________________________________

## 📝 Citation

If you find Relax useful in your research, please cite:

```bibtex
@software{relax2026,
  title  = {Relax: An Asynchronous Reinforcement Learning Engine for Omni-Modal Post-Training at Scale},
  author = {Relax Contributors},
  url    = {https://arxiv.org/abs/2604.11554},
  year   = {2026}
}
```

______________________________________________________________________

## 📜 License

This project is licensed under the [Apache License 2.0](./LICENSE).

______________________________________________________________________

## 🙏 Acknowledgements

Relax is built upon the shoulders of excellent open-source projects:

- [Slime](https://github.com/THUDM/slime) — Scalable training and inference framework for reinforcement learning
- [SGLang](https://github.com/sgl-project/sglang) — Fast serving framework for large language models
- [Megatron-LM](https://github.com/NVIDIA/Megatron-LM) & [Megatron-Bridge](https://github.com/NVIDIA-NeMo/Megatron-Bridge) — Large-scale distributed training framework and HF ↔ Megatron weight conversion bridge, with sincere thanks to the entire **NVIDIA** team
- [TransferQueue](https://github.com/Ascend/TransferQueue) — High-performance distributed data transfer queue
- [Ray](https://github.com/ray-project/ray) — Distributed computing framework
- [HuggingFace Transformers](https://github.com/huggingface/transformers) — State-of-the-art model hub

We sincerely thank all contributors and the open-source community for making this project possible.

Relax

다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Relax

다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README