KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #typescript #ai-agent #developer-tools #claude-code #codex #deepseek-harness #ai-agents #react #mcp #rust

cambrian-s

★ 565 Open GitHub ↗

Cambrian-S: Towards Spatial Supersensing in Video

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# <img src="figs/tesseract.png" alt="Cambrian-S" width="28" height="28" style="vertical-align: middle;"> *Cambrian-S*:<br> Towards Spatial Supersensing in Video

<a href="https://arxiv.org/abs/2511.04670" target="_blank">
    <img alt="arXiv" src="https://img.shields.io/badge/arXiv-Cambrian--S-red?logo=arxiv" height="25" />
</a>
<a href="https://cambrian-mllm.github.io/cambrian-s/" target="_blank">
    <img alt="Website" src="https://img.shields.io/badge/🌎_Website-cambrian--mllm.github.io-blue" height="25" />
</a>
<a href="https://huggingface.co/collections/nyu-visionx/cambrian-s-models" target="_blank">
    <img alt="HF Model: Cambrian-S" src="https://img.shields.io/badge/%F0%9F%A4%97%20_Model-Cambrian--S-ffc107?color=ffc107&logoColor=white" height="25" />
</a>
<a href="https://huggingface.co/datasets/nyu-visionx/VSI-590K" target="_blank">
    <img alt="HF Dataset: VSI-590K" src="https://img.shields.io/badge/%F0%9F%A4%97%20_Data-VSI--590K-ffc107?color=ffc107&logoColor=white" height="25" />
</a>
<a href="https://huggingface.co/collections/nyu-visionx/vsi-super" target="_blank">
    <img alt="HF Dataset: VSI-Super" src="https://img.shields.io/badge/%F0%9F%A4%97%20_Benchmark-VSI--Super-ffc107?color=ffc107&logoColor=white" height="25" />
</a>
<div style="font-family: charter;">
    <a href="https://github.com/vealocia" target="_blank">Shusheng Yang*</a>,
    <a href="https://jihanyang.github.io/" target="_blank">Jihan Yang*</a>,
    <a href="https://pinzhihuang.github.io/" target="_blank">Pinzhi Huang†</a>,
    <a href="https://ellisbrown.github.io/" target="_blank">Ellis Brown†</a>,
    <a href="https://redagavin.github.io/" target="_blank">Zihao Yang</a>,
    <br>
    <a href="https://github.com/czyuyue" target="_blank">Yue Yu</a>,
    <a href="https://tsb0601.github.io/" target="_blank">Shengbang Tong</a>,
    <a href="#" target="_blank">Zihan Zheng</a>,
    <a href="https://www.yfxu.com/" target="_blank">Yifan Xu</a>,
    <a href="https://github.com/wang-muhan" target="_blank">Muhan Wang</a>,
    <a href="https://daohanlu.github.io/" target="_blank">Daohan Lu</a>,
    <br>
    <a href="https://cs.nyu.edu/~fergus/pmwiki/pmwiki.php" target="_blank">Rob Fergus</a>,
    <a href="http://yann.lecun.com/" target="_blank">Yann LeCun</a>,
    <a href="https://profiles.stanford.edu/fei-fei-li" target="_blank">Li Fei-Fei</a>,
    <a href="https://www.sainingxie.com/" target="_blank">Saining Xie</a>
</div>

<div style="font-family: charter;">
*Equal Contribution&nbsp;&nbsp;&nbsp;&nbsp;†Core Contributor
</div>

</div>

## Release
- [Apr 3, 2026] 🚀 Meta information of our curated VSI-590K dataset is released here [VSI-590K-MetaInfo](https://huggingface.co/datasets/nyu-visionx/VSI-590K-MetaInfo).
- [Dec 21, 2025] 🚀 [Cambrian-S-3M](https://huggingface.co/datasets/nyu-visionx/Cambrian-S-3M) (our collection of 3M open-sourced video instruction tuning data) is now available! Please check it out!
- [Nov 6, 2025] 🔥 We release Cambrian-S model weights, training code, and evaluation suite.
- [Nov 6, 2025] 🔥 We release VSI-SUPER, a benchmark designed for spatial supersensing.
- [Nov 6, 2025] 🔥 We release VSI-590K, a dataset curated for spatial sensing.

## Contents
- [ *Cambrian-S*: Towards Spatial Supersensing in Video](#-cambrian-s-towards-spatial-supersensing-in-video)
  - [Release](#release)
  - [Contents](#contents)
  - [Cambrian-S Weights](#cambrian-s-weights)
    - [General Model Performance](#general-model-performance)
    - [VSI-SUPER Performance](#vsi-super-performance)
    - [Model Card](#model-card)
      - [Model Trained with Predictive Sensing](#model-trained-with-predictive-sensing)
      - [Standard MLLM Models](#standard-mllm-models)
  - [VSI-590K Dataset](#vsi-590k-dataset)
  - [Train](#train)
  - [Evaluation](#evaluation)
  - [Citation](#citation)
  - [Related Projects](#related-projects)

## Cambrian-S Weights

Here are our Cambrian-S checkpoints along with instructions on how to use the weights. Our models excel at spatial reasoning in video understanding, demonstrating significant improvements over previous state-of-the-art methods on spatial understanding benchmarks while maintaining competitive performance on general video understanding tasks.

### General Model Performance

Comparison of Cambrian-S with other leading MLLMs on general video understanding benchmarks.

**Results**: Cambrian-S maintains competitive performance on standard video benchmarks (Perception Test and EgoSchema) while excelling at spatial reasoning tasks.

### VSI-SUPER Performance

VSI-SUPER performance is evaluated on **Cambrian-S-7B-LFP**.

### Model Card

#### Model Trained with Predictive Sensing

#### Standard MLLM Models

| Model           | Base-LLM | Vision Encoder | Hugging Face                                                    |
|-----------------|------------|----------------|------------------------------------------------------------------|
| Cambrian-S-7B   | `Qwen2.5-7B-Instruct`         | `siglip2-so400m-patch14-384`     | [nyu-visionx/Cambrian-S-7B](https://huggingface.co/nyu-visionx/Cambrian-S-7B)   |
| Cambrian-S-3B   | `Qwen2.5-3B-Instruct`         | `siglip2-so400m-patch14-384`     | [nyu-visionx/Cambrian-S-3B](https://huggingface.co/nyu-visionx/cambrian-s-3b)   |
| Cambrian-S-1.5B | `Qwen2.5-1.5B-Instruct`       | `siglip2-so400m-patch14-384`     | [nyu-visionx/Cambrian-S-1.5B](https://huggingface.co/nyu-visionx/cambrian-s-1.5b) |
| Cambrian-S-0.5B | `Qwen2.5-0.5B-Instruct`       | `siglip2-so400m-patch14-384`     | [nyu-visionx/Cambrian-S-0.5B](https://huggingface.co/nyu-visionx/cambrian-s-0.5b) |

## VSI-590K Dataset

VSI-590K is a video instruction-tuning dataset focusing on spatial understanding.

**VSI-590K dataset statistics.**

QAs are grouped by: question types (left) and task groups (right).

**Hugging Face**: [nyu-visionx/VSI-590K](https://huggingface.co/datasets/nyu-visionx/vsi-590k)

## Train

### Environment Preparation

Currently, we support training on TPU using TorchXLA. Install `TorchXLA 2.6.0` by the following commands:

```bash
pip install torch==2.6.0 torchvision==0.21.0 torch_xla==2.6.0
pip install 'torch_xla[tpu]' -f https://storage.googleapis.com/libtpu-releases/index.html
pip install 'torch_xla[pallas]' -f https://storage.googleapis.com/jax-releases/jax_nightly_releases.html -f https://storage.googleapis.com/jax-releases/jaxlib_nightly_releases.html
pip install --upgrade pip
pip install -e '.[tpu]'
```

### Data Preparation

Cambrian-S models are trained on top of [`Cambrian-Alignment`](https://huggingface.co/datasets/nyu-visionx/Cambrian-Alignment), [`Cambrian-7M`](https://huggingface.co/datasets/nyu-visionx/Cambrian-10M), [`Cambrian-S-3M`](https://huggingface.co/datasets/nyu-visionx/Cambrian-S-3M), and [`VSI-590K`](https://huggingface.co/datasets/nyu-visionx/VSI-590K) datasets. Please prepare these datasets following their corresponding guidelines.

### Training Scripts

As mentioned in our paper, Cambrian-S models are trained in 4 stages: from vision-language alignment, to general image instruction tuning, and general video instruction tuning, and finally spatial video tuning. For Cambrian-S-LFP model, we modified the 4th stage by involving latent frame prediction objective. We provides sample training scripts in the following:

* [cambrian/scripts/cambrians_7b_s1.sh](cambrian/scripts/cambrians_7b_s1.sh)
* [cambrian/scripts/cambrians_7b_s2.sh](cambrian/scripts/cambrians_7b_s2.sh)
* [cambrian/scripts/cambrians_7b_s3.sh](cambrian/scripts/cambrians_7b_s3.sh)
* [cambrian/scripts/cambrians_7b_s4.sh](cambrian/scripts/cambrians_7b_s4.sh)
* [cambrian/scripts/cambrians_7b_lfp_s4.sh](cambrian/scripts/cambrians_7b_lfp_s4.sh)

## Evaluation

We have released our evaluation code in the [`lmms-eval/`](lmms-eval/) subfolder. Please see the README there for more details.

For detailed benchmark results, please refer to the [General Model Performance](#general-model-performance) and [VSI-SUPER Performance](#vsi-super-performance) sections above.

## Citation

If you find our work useful for your research, please consider to cite our work:

```bibtex
@article{yang2025cambrians,
  title={Cambrian-S: Towards Spatial Supersensing in Video},
  author={Yang, Shusheng and Yang, Jihan and Huang, Pinzhi and Brown, Ellis and Yang, Zihao and Yu, Yue and Tong, Shengbang and Zheng, Zihan and Xu, Yifan and Wang, Muhan and Lu, Daohan and Fergus, Rob and LeCun, Yann and Fei-Fei, Li and Xie, Saining},
  journal={arXiv preprint arXiv:2511.04670},
  year={2025}
}

@article{brown2025shortcuts,
  author = {Brown, Ellis and Yang, Jihan and Yang, Shusheng and Fergus, Rob and Xie, Saining},
  title = {Benchmark Designers Should ``Train on the Test Set'' to Expose Exploitable Non-Visual Shortcuts},
  journal = {arXiv preprint arXiv:2511.04655},
  year = {2025}
}

@article{brown2025simsv,
  title   =  { {SIMS-V}: Simulated Instruction-Tuning for Spatial Video Understanding },
  author  =  { Brown, Ellis and Ray, Arijit and Krishna, Ranjay and Girshick, Ross and Fergus, Rob and Xie, Saining },
  journal =  { arXiv preprint arXiv:2511.04668 },
  year    =  { 2025 }
}

@article{yang2024think,
    title={{Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces}},
    author={Yang, Jihan and Yang, Shusheng and Gupta, Anjali W. and Han, Rilyn and Fei-Fei, Li and Xie, Saining},
    year={2024},
    journal={arXiv preprint arXiv:2412.14171},
}

@article{tong2024cambrian,
  title={{Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs}},
  author={Tong, Shengbang and Brown, Ellis and Wu, Penghao and Woo, Sanghyun and Middepogu, Manoj and Akula, Sai Charitha and Yang, Jihan and Yang, Shusheng, and Iyer, Adithya and Pan, Xichen and Wang, Austin and Fergus, Rob and LeCun, Yann and Xie, Saining},
  journal={arXiv preprint arXiv:2406.16860},
  year={2024}
}
```

## Related Projects

- [Cambrian-1](https://github.com/cambrian-mllm/cambrian): A Fully Open, Vision-Centric Exploration of Multimodal LLMs
- [Thinking in Space](https://vision-x-nyu.github.io/thinking-in-space.github.io/): How Multimodal Large Language Models See, Remember and Recall Spaces - Introduces VSI-Bench for evaluating visual-spatial intelligence
- [SIMS-V](https://ellisbrown.github.io/sims-v): Simulated Instruction-Tuning for Spatial Video Understanding
- [Test-Set Stress-Test](https://vision-x-nyu.github.io/test-set-training): Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts

cambrian-s

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README