turnstile

KO | EN

gitlite — search

#hacktoberfest #react #typescript #llm #android #machine-learning #deep-learning #linux #macos #windows #awesome-list #security #openai #open-source

(★ 21)

No description available.

turnstile Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Turnstile

**Token-level rollout capture for RL training of LLM agents.**

Turnstile is a proxy that sits between an agent orchestrator and an inference backend. To the orchestrator it looks like a normal OpenAI Chat Completions API. To the trainer, it exposes token sequences, trainable token segments, and log-probabilities you actually need to update a model. It closes the gap between agent rollout frameworks and RL training frameworks without requiring either side to change.

- **Drop-in proxy.** Any orchestrator that speaks the OpenAI Chat Completions API works — [Strands](https://github.com/strands-agents/sdk-python), [OpenHands](https://github.com/All-Hands-AI/OpenHands), or your own harness. No SDK changes or special clients needed.
- **Token-level capture.** Records exact prompt/output token IDs and per-token logprobs straight from the backend, so training sees the same tokens the model saw.
- **Prefix-aware sequence collapsing.** Multi-turn agent loops collapse into one contiguous sequence with output/input segment metadata instead of N redundant copies of the history.
- **Generic training schema.** Exports `(tokens, segment_info, logprobs)`. Framework-specific conversion (SLIME, etc.) lives in client-side adapters, not in the proxy.
- **Pluggable frontend and backend.** OpenAI Chat Completions on the front, sglang on the back today — both interfaces are designed to grow.
- **Embeddable.** Run as a standalone server, or drop it into a Python training script via PyO3 bindings and get in-process access to training data.

## Problem

RL training for LLM agents splits into two phases handled by different systems:

1. **Rollout** — an agent attempts a task. Orchestrators like [Strands](https://github.com/strands-agents/sdk-python), [OpenHands](https://github.com/All-Hands-AI/OpenHands), and others drive this through an OpenAI-compatible chat API.
2. **Training** — a model is updated on what happened. Frameworks like [SLIME](https://github.com/huggingface/slime) need token sequences, model-output boundaries, and log-probabilities.

The orchestrator talks to a black-box chat API. The trainer needs the stuff that black box hides. The agent doesn't know — or need to know — the exact tokens its model saw, but the trainer can't update weights without them. Teams end up writing custom adapters, forking orchestrators, or piping brittle log scraping into training — and nobody owns the gap. Turnstile owns it, so neither the orchestrator nor the trainer has to change.

## What Turnstile does

Turnstile sits in the middle. The orchestrator thinks it's talking to a normal OpenAI API. Turnstile forwards requests to the real inference backend (sglang today, vLLM and others planned), streams responses back, and silently records every token in and out along with its logprob.

```mermaid
flowchart LR
    Orchestrator["Orchestrator\n(any OpenAI-compatible)"]
    Turnstile["Turnstile\n(proxy)"]
    Engine["Inference Engine\n(sglang, vllm)"]
    Data["Training-ready\nsequences + segments"]
    Training["RL Training\n(SLIME, etc.)"]

Orchestrator -- "OpenAI API" --> Turnstile
    Turnstile -- "stream response" --> Orchestrator
    Turnstile -- "Backend API" --> Engine
    Engine -- "stream response" --> Turnstile
    Turnstile -- "episode complete" --> Data
    Data --> Training
```

When the rollout finishes, the training loop asks Turnstile for the episode's data and gets back training-ready sequences grouped by `group_id` (an arbitrary user-chosen tag for a rollout or experiment).

## Quickstart

Install the Python bindings:

```bash
pip install turnstile
```

Embed Turnstile inside a training script and route an agent through it:

```python
import turnstile
from openai import OpenAI

with turnstile.Proxy(
    backend_url="http://localhost:30000",        # your sglang server
    model="Qwen/Qwen2.5-0.5B-Instruct",
) as proxy:
    group_id = proxy.create_group(max_trajectory_tokens=4096)

# Point any OpenAI-compatible orchestrator at proxy.addr.
    # group_id is supplied via the URL path: /group/<id>/v1/...
    client = OpenAI(base_url=f"http://{proxy.addr}/group/{group_id}/v1", api_key="-")

# ... run your agent against `client`; Turnstile captures every turn ...
    client.chat.completions.create(
        model="Qwen/Qwen2.5-0.5B-Instruct",
        messages=[{"role": "user", "content": "Plan a trip to Paris."}],
    )

# At the end of the episode, pull training data in-process.
    for seq in proxy.get_training_sequences(group_id):
        tokens   = seq.tokens     # list[int]
        segments = seq.segment_info  # list[tuple[bool, int]] — (is_output, length)
        logprobs = seq.logprobs   # list[float] — 0.0 at input positions
        versions = seq.weight_versions  # list[tuple[int, str]]
```

Any orchestrator with a configurable base URL (Strands, OpenHands, raw OpenAI SDK, etc.) can point at Turnstile with a one-line config change after creating a group for the run. The agent code is unchanged between an unmonitored run and a training run.

Prefer a standalone server? Run `turnstile` as a service, create a group with `POST /v1/create_group`, and talk to it over HTTP with the `turnstile` Python client or any HTTP client. Training data is fetched with `GET /group/{id}/v1/training/sequences`.

## Sequence construction

Turnstile handles the two shapes agent rollouts actually take:

**Prefix-overlapping turns (the common multi-turn agent shape).** Each turn's prompt is the full conversation history plus new environment tokens, so turn N+1's input is a suffix extension of turn N's full sequence. Turnstile detects the overlap and collapses every turn into one contiguous sequence with segment metadata:

```
tokens: [sys] [usr] [ast] [ast] [ast] [env] [env] [ast] [ast] [ast] [env] [ast] [ast] ...
segment_info: [[false, 2], [true, 3], [false, 2], [true, 3], [false, 1], [true, 2], ...]
```

`true` segments are model-generated and trainable; `false` segments are environment/user input. This eliminates the N redundant copies of the conversation prefix you'd get from recording each turn independently, and produces exactly the shape an RL trainer expects.

**Non-overlapping turns.** When turns don't share a prefix — independent requests within an episode, or an orchestrator that rewrites context between turns — collapsing isn't possible. Turnstile returns multiple independent sequences per episode, each with its own segment metadata, and the trainer treats them as separate examples.

## Training data schema

One schema for every backend and every trainer:

- **`tokens`** — the full collapsed token sequence for the episode (or sequence segment).
- **`segment_info`** — run-length encoded `[is_output, length]` pairs, e.g. `[[false, 41], [true, 12]]`.
- **`logprobs`** — per-token log-probabilities; `0.0` at input positions, actual backend values at generated positions.
- **`weight_versions`** — token offsets where SGLang reported a new backend weight version for the turn, e.g. `[[0, "default"], [41, "2"]]`.
- **`routed_experts`** — optional base64-encoded SGLang MoE routing payload for R3 routing replay. Enable it with `TURNSTILE_RETURN_ROUTED_EXPERTS=true` or `turnstile.Proxy(..., return_routed_experts=True)`. By default, `TURNSTILE_ROUTED_EXPERTS_MODE=strict` starts a new sequence when routed experts for an overlapping prefix disagree; `latest` keeps the sequence collapsed and exports the latest full payload. In Python, pass `turnstile.RoutedExpertsMode.STRICT` or `turnstile.RoutedExpertsMode.LATEST` for `routed_experts_mode`; raw strings are rejected by the Python bindings. Downstream training decodes it as int32 expert IDs and reshapes it with the model's `(num_layers, top_k)`.
- **`response_parser`** — controls how decoded backend output becomes assistant content and structured tool calls. The default `qwen_permissive` preserves Turnstile's liberal parser. Set `TURNSTILE_RESPONSE_PARSER=qwen_sglang` or `turnstile.Proxy(..., response_parser="qwen_sglang")` for SGLang-compatible Qwen non-stream tool-call parsing.

Turnstile always sends the group ID as the SGLang Model Gateway routing header. SGLang only uses that header for worker affinity when the request goes through a gateway configured for consistent-hashing routing; direct worker requests and other routing modes ignore it. If a deployment also needs separate SGLang radix-cache namespaces per group, enable `TURNSTILE_SGLANG_GROUP_CACHE_NAMESPACE=true` or `turnstile.Proxy(..., sglang_group_cache_namespace=True)`; this sends the group ID as SGLang's backend-only JSON `extra_key` field. The cache namespace option is disabled by default and requires SGLang v0.5.11 or newer; older releases accepted `extra_key` but did not pass it into scheduler prefix-cache matching. See https://github.com/sgl-project/sglang/pull/23300.

Framework-specific conversion (e.g. SLIME's expected layout) lives in client-side adapters, not in the proxy. The proxy stays small and generic.

### R3 MoE routing replay

R3 capture is disabled by default and is not part of CI. See
[docs/r3-moe-smoke-test.md](docs/r3-moe-smoke-test.md) for the manual smoke
test against a real MoE model.

## Architecture

Three components, intentionally small:

- **Core proxy (Rust).** Async Axum server. Handles request forwarding, response streaming, token recording, sequence collapsing, and group-scoped storage. Loads the tokenizer and HuggingFace chat template directly so that what the trainer sees matches what the model saw.
- **Python bindings (PyO3).** Wrap the Rust core. Start/stop the proxy from a training script, read training data in-process — no extra HTTP hop when you're co-located.
- **Python client library.** Standalone HTTP client for remote Turnstile instances, plus adapters that convert the generic schema into framework-specific formats (SLIME first; more to follow).

The API surface is modular on purpose. The OpenAI Chat Completions frontend is what most orchestrators already speak, and the recording and collapsing logic is independent of any one API shape. The inference backend interface operates at the token level (token-in, token-out), so adding vLLM or another token-level engine doesn't touch capture or collapsing.

## Ecosystem

Designed to compose with what people are already using:

| Layer              | Works with today           | Designed to add |
| ------------------ | -------------------------- | --------------- |
| Orchestrator       | [Strands](https://github.com/strands-agents/sdk-python), [OpenHands](https://github.com/All-Hands-AI/OpenHands), anything using the OpenAI Chat Completions API | — |
| Inference backend  | sglang                     | vLLM, other token-level engines |
| Training framework | SLIME (via client adapter) | Additional adapters live in the client library |

Turnstile is orchestrator-agnostic by design — any system that speaks OpenAI Chat Completions can produce RL-ready rollouts through it without modification.

## What Turnstile is not

To keep the scope honest:

- Not a model, dataset, or set of weights.
- Not an RL algorithm or training recipe — it captures rollouts, it doesn't prescribe how you train on them.
- Not a replacement for your orchestrator, inference engine, or training framework.
- Not a managed or production service.

## Security posture

Turnstile has no built-in authentication, authorization, TLS termination, or
multi-tenant isolation. Run it only on trusted networks or behind external
access controls that match your deployment requirements.

## Status

Early development. APIs, schemas, and backend support will move. Feedback and integrations are welcome.

turnstile

turnstile Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

turnstile

turnstile Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README