fal-worldclaw

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

★ 21 Open GitHub ↗

No description available.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# WorldClaw on fal

**One open-ended prompt in. An explorable, editable 3D world out.**

A browser reimplementation of [*WorldClaw: Agentic 3D Open-World Generation at Scale*](https://arxiv.org/abs/2608.05248)
by the Tencent Hunyuan3D team, rebuilt to run end to end on [fal](https://fal.ai) generative endpoints.

[![arXiv](https://img.shields.io/badge/arXiv-2608.05248-b31b1b)](https://arxiv.org/abs/2608.05248)
[![Powered by fal](https://img.shields.io/badge/powered%20by-fal.ai-e8a33d)](https://fal.ai)
[![Next.js](https://img.shields.io/badge/Next.js-16-black)](https://nextjs.org)
[![three.js](https://img.shields.io/badge/three.js-r182-049EF4)](https://threejs.org)

*"A medieval-style village scene with diverse terrain, including snow-capped mountains, plains, bodies of water, and a desert, populated with animals." One prompt, one run, a few minutes, and roughly fifty cents of fal credits.*

</div>

## What this is

The [WorldClaw paper](https://arxiv.org/abs/2608.05248) (Tencent Hunyuan3D, 2026) describes a fully agentic, coarse-to-fine framework for open-world 3D scene generation. Planning agents translate a text prompt into a structured specification of regions, terrain, assets, materials, and spatial relations. The system then builds a globally coherent terrain, populates it with editable instance-level 3D assets, and refines the result by looking at its own renders. The original runs on a GPU cluster and orchestrates Claude, GPT-Image, SAM 3D, Hunyuan3D, and Blender through MCP.

This project asks a simple question: how much of that pipeline can run today, in a browser tab, with nothing but a fal API key?

The answer turned out to be nearly all of it. Every model in the loop, from the planning LLM to image generation, segmentation-based 3D reconstruction, seamless PBR materials, and the vision model that critiques the finished scene, is a fal endpoint. The deterministic parts (height fields, placement solving, procedural scatter geometry) run as TypeScript in the page. The result is a live, walkable, editable world rendered in three.js.

*The isometric overview, mirroring the layout renders from the paper's gallery.*
</div>

## The pipeline

Each generation walks the paper's coarse-to-fine loop, and you watch it happen stage by stage in the agent timeline:

| Stage | What happens | Powered by |
|---|---|---|
| 1. Intent analysis | Extracts only the constraints your prompt states explicitly, inventing nothing | `openrouter/router` (Claude Sonnet) |
| 2. Scene planning | An agent authors the full world spec: regions, landforms, materials, assets, placement rules | `openrouter/router` |
| 3. Terrain foundation | Domain-warped region masks compose a region-aware height field with rivers, shores, and micro-relief | Deterministic TypeScript, seeded |
| 4. Placement solve | Every instance gets a terrain-aligned transform: clusters, rows, rings, waterfronts, slope limits, spacing | Deterministic TypeScript, seeded |
| 5. Surface materials | One genuinely seamless tiling material per region, sampled at native resolution by a splat shader | `fal-ai/patina/material` |
| 6. Asset prototypes | Concept image, then image-to-3D reconstruction into a textured GLB, normalized and cloned per instance | `fal-ai/flux/schnell` + `fal-ai/sam-3/3d-objects` or `fal-ai/hunyuan-3d/v3.1/pro/image-to-3d` |
| 7. Render critique | The app screenshots its own canvas; a vision agent inspects the renders and orders corrections | `openrouter/router/vision` |

Trees, rocks, and grass come from a seeded procedural library (the paper's "3D coding" pathway for scatter assets), so meadows and forests stay dense at zero marginal cost.

*Mid-generation: the terrain is already explorable while materials and meshes stream in from the fal queue.*
</div>

## The four render channels

The paper renders every world four ways: appearance, instance masks, surface normals, and depth. So does the viewport, live, at the flick of a toggle. The instance-mask channel doubles as a picker: every object in the world is a separate, selectable, editable instance.

<div align="center">
<table>
<tr>
<td><img src="docs/shots/channel-instances.png" alt="Instance mask channel" width="440"></td>
<td><img src="docs/shots/channel-normals.png" alt="Surface normals channel" width="440"></td>
</tr>
<tr>
<td align="center"><em>Instance masks</em></td>
<td align="center"><em>Surface normals</em></td>
</tr>
</table>
</div>

## Why fal

This is, deliberately, a stress test of building a multi-model agentic system on a single platform:

- **One API key, five model families.** LLM routing (Claude, Gemini, and 200+ models through `openrouter/router`), FLUX image generation, Patina seamless materials, SAM-3D object reconstruction, and Hunyuan 3D 3.1 Pro all speak the same queue protocol.
- **The queue absorbs the burst.** A single generation fires around twenty jobs at once. Nothing is rejected; jobs wait for concurrency slots and the app just subscribes to updates.
- **Outputs chain without plumbing.** Every result is a CDN URL that feeds directly into the next endpoint as input. The concept image goes straight into the 3D reconstructor; canvas screenshots upload to fal storage and go straight into the vision critique.
- **Swapping a stage is one endpoint id.** The mesh engine toggle in the UI is literally that: the same pipeline pointed at SAM-3D (fast, about $0.02 per asset) or Hunyuan 3D 3.1 Pro (film-grade PBR geometry).
- **The key never touches the browser.** All client traffic flows through [`@fal-ai/server-proxy`](https://docs.fal.ai) with an endpoint allowlist.

A full world on the default settings costs roughly **$0.40 to $0.70** and takes a few minutes, most of it queue time.

## Quickstart

```bash
git clone https://github.com/blendi-remade/fal-worldclaw
cd fal-worldclaw
npm install
cp .env.example .env.local   # add your FAL_KEY from https://fal.ai/dashboard/keys
npm run dev
```

Open http://localhost:3000, pick one of the eleven prompts from the paper's gallery (or write your own), and hit Generate.

No key yet? The landing page has an **offline demo world**: the full deterministic pipeline with procedural assets, zero network calls.

## Exploring and editing

| | |
|---|---|
| Camera | Orbit, isometric overview, or first-person walk (WASD, Shift to sprint) |
| Channels | Appearance, instance masks, normals, depth |
| Editing | Click any object to select it; move, rotate, scale with gizmos; duplicate or delete; every instance of a prototype rescales from the inspector |
| Materials | Per-region tile size sliders re-skin the ground instantly |
| Zoom | Click a prototype card in the inspector and the camera flies to one of its instances |
| Export | Download the full world spec plus every instance transform and asset URL as JSON |

## Fidelity to the paper

Faithful where it counts:

- The three-stage global-to-regional architecture (plan, terrain, populate) with agents communicating through a shared structured specification
- Strictly extractive intent analysis, exactly as Section 2.1 defines it
- Region-aware height field built from soft semantic masks with per-kind geomorphic profiles (Eq. 6 in spirit: base elevation plus weighted noise plus terrace, dune, peak, and crater operators)
- Terrain-aligned placement transforms for every object, with independent per-instance editability
- Render-based refinement: diagnostic renders from fixed viewpoints, a critique agent, applied corrections
- The four diagnostic render channels, the isometric overview, and the ground-level walk

Deviations, honestly noted: placements come from the planning agent plus a deterministic solver rather than the paper's terrain-conditioned composition images with SAM3 segmentation and camera-ray placement recovery (Eq. 9 to 13); materials are seamless tiles blended by a splat shader rather than Blender node graphs; refinement applies a fixed vocabulary of corrections rather than free-form engine edits. And it fits in a browser tab.

## Acknowledgements

- **The WorldClaw paper and the Tencent Hunyuan3D team**, whose work this project reimplements: [arXiv:2608.05248](https://arxiv.org/abs/2608.05248), [project page](https://tencent-hunyuan.github.io/Hunyuan3D-WorldClaw/). The eleven preset prompts are the prompts from their gallery. The quality-mode mesh engine is their own Hunyuan 3D 3.1 Pro.
- **[fal](https://fal.ai)** for hosting every model in the loop behind one API: FLUX, Patina, SAM-3D, Hunyuan 3D, and the LLM router.
- Meta's SAM 3D, Black Forest Labs' FLUX, and the three.js and React Three Fiber communities.

Built as an exploration of what agentic 3D generation looks like when the entire model stack is a single platform away.

fal-worldclaw

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README