KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #developer-tools #ai-agent #claude-code #codex #typescript #ai-agents #deepseek-harness #mcp #cli #rust

RINO

★ 46 Open GitHub ↗

Official PyTorch implementation of "Let RGB Be the Language of Vision".

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# RINO: RGB In, RGB Out

Official PyTorch implementation of "Let RGB Be the Language of Vision".

**Authors:** Timing Yang, Jinrui Yang, Xinlong Li, Yuhan Wang, Haoran Li, Yanqing Liu, Guoyizhe Wei, Jixuan Ying, Chen Wei, Rama Chellappa, Yuyin Zhou, Cihang Xie, Alan Yuille, Feng Wang\*

Johns Hopkins University · UC Santa Cruz · Carnegie Mellon University · Rice University

\*Project lead

[![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b.svg)](https://arxiv.org/pdf/2607.12450)
[![Project Page](https://img.shields.io/badge/Project-Page-2ea44f.svg)](https://yangtiming.github.io/RINO_Page/)

<img src="assets/teaser.png" width="100%"
 alt="RINO unifies vision under a single RGB interface: one frozen image editor, driven by a task-specific prompt, handles estimation (depth, normals, segmentation, detection, pose, referring) and the corresponding conditioned generation.">

RINO renders every input and output as RGB, so one frozen image editor handles
both estimation and conditioned generation — no task-specific module.

Zero-shot vision benchmarks for open-source image-edit models under a single
RGB-to-RGB interface. Every task is expressed as image editing: understanding
tasks render the structured output (mask, depth, normals, pose, …) as an RGB
image, and conditioned-generation tasks consume an RGB condition. No
task-specific head, encoder, decoder, or adapter is added — the editor is used
as a frozen black box.

Each task lives under [`tasks/`](tasks/) and follows the same layout —
`tasks/<task>/evaluate/` for code, `tasks/<task>/output/` for runtime results —
and is scored by **copied official metric code** so the numbers are comparable
to the literature. Shared image-edit backends live at the repo root in
[`core/`](core/). See each task's README for full details.

## Backends

Three open-source image editors, all used as a black-box RGB-to-RGB editor
(released weights, no fine-tuning), wired up in [`core/`](core/):

- Qwen-Image-Edit — `Qwen/Qwen-Image-Edit-2511`
- FireRed-Image-Edit — `FireRedTeam/FireRed-Image-Edit-1.0`
- LongCat-Image-Edit — `meituan-longcat/LongCat-Image-Edit`

Every task accepts `BACKEND=qwen|firered|longcat` with a matching `MODEL=<hf-id>`.

## Demo

Two ready-to-run scripts under [`demo/`](demo/) show the RGB-in / RGB-out
interface end-to-end — one estimation task, one conditioned-generation task.
No dataset, no metrics; defaults chain together so both run with zero args.

```bash
pip install -r requirements.txt

# 1) RGB -> grayscale depth
python demo/estimation_depth.py
# writes demo_output/depth/raw.png  (grayscale depth: near=bright, far=dark)
#        demo_output/depth/pred.npy (relative depth)

# 2) depth + caption -> RGB scene (consumes step 1's output)
python demo/generation_depth.py
# writes demo_output/gen_depth/scene.png
```

<img src="assets/demo_chain.png" width="100%" alt="RINO demo chain">

The two commands above, run back-to-back: one frozen editor turns an RGB photo into a grayscale depth map (step 1), then re-hallucinates an RGB scene from that depth (step 2).

Point them at your own inputs with `--image`, `--depth`, `--caption`, or
override the editor with `--backend qwen|firered|longcat` + a matching
`--model <hf-id>` (default: `Qwen/Qwen-Image-Edit-2511`). Weights are pulled
from Hugging Face on first run. For the full benchmark on each task see
[`tasks/<task>/README.md`](tasks/).

## Citation

```bibtex
@misc{yang2026letrgblanguagevision,
      title={Let RGB Be the Language of Vision}, 
      author={Timing Yang and Jinrui Yang and Xinlong Li and Yuhan Wang and Haoran Li and Yanqing Liu and Guoyizhe Wei and Jixuan Ying and Chen Wei and Rama Chellappa and Yuyin Zhou and Cihang Xie and Alan Yuille and Feng Wang},
      year={2026},
      eprint={2607.12450},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2607.12450}, 
}
```

RINO

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README