vlm-world-model

KO | EN

gitlite — search

#javascript #llm #go #machine-learning #deep-learning #macos #awesome #cli #kubernetes #vue #awesome-list #ios #openai #open-source

vlm-world-model

(★ 11)

Materials for paper "Bootstrapping World Models from Dynamics Models in Multimodal Foundation Models"

vlm-world-model 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 style="text-align: center;">Bootstrapping World Models from Dynamics Models in Multimodal Foundation Models</h1>

<div>
  <sup>1</sup><a href='https://yfqiu.netlify.app/' target='_blank'><b>Yifu Qiu</b></a>&emsp;
  <sup>3</sup><a href='https://scholar.google.co.il/citations?user=37SMCrsAAAAJ&hl=iw' target='_blank'>Yftah Ziser</b></a>&emsp;
  <sup>2</sup><a href='https://scholar.google.co.uk/citations?user=SCoVoOYAAAAJ&hl=en' target='_blank'>Anna Korhonen</b></a>&emsp;
  <sup>1</sup><a href='https://scholar.google.com/citations?user=Q4oVM7IAAAAJ&hl=en' target='_blank'>Shay B. Cohen</b></a>&emsp;
  <sup>1,2,3</sup><a href='https://ducdauge.github.io/' target='_blank'>Edoardo M. Ponti</b></a>&emsp;

</div>
<div><sup>1</sup>ILCC, University of Edinburgh</div>
<div><sup>2</sup>Language Technology Lab, University of Cambridge</div>
<div><sup>3</sup>NVIDIA</div>

![](https://img.shields.io/badge/PRs-welcome-brightgreen) 
[![Website](https://img.shields.io/badge/Website-AURORA.svg)](https://yfqiu-nlp.github.io/vlm-world-model.github.io/)
[![arxiv](https://img.shields.io/badge/arXiv-2506.06006-b31b1b.svg)](https://arxiv.org/abs/2506.06006)
[![HF Model: CWM](https://img.shields.io/badge/HF%20Model-CWM-FFD21E.svg)](https://huggingface.co/yfqiu-nlp/chameleon-world-model-aurora)

</h4>
</div>

![Demo GIF](https://raw.githubusercontent.com/yfqiu-nlp/vlm-world-model/main/figures/vlm-world-model.gif)

## Abstract
To what extent do vision-and-language foundation models possess a realistic world model (observation × action → observation) and a dynamics model (observation × observation → action), when actions are expressed through language? While open-source foundation models struggle with both, we find that fine-tuning them to acquire a dynamics model through supervision is significantly easier than acquiring a world model. In turn, dynamics models can be used to bootstrap world models through two main strategies: 1) weakly supervised learning from synthetic data and 2) inference time verification. Firstly, the dynamics model can annotate actions for unlabelled pairs of video frame observations to expand the training data. We further propose a loss-weighting mechanism for the image tokens weighted by the its importance predicted by a recognition model. Secondly, the dynamics models can assign rewards to multiple samples of the world model to score them, effectively guiding search at inference time. We evaluate the world models resulting from both strategies through the task of action-centric image editing on AURORA-BENCH. Our best model achieves a performance competitive with state-of-the-art image editing models, improving on them by a margin of 15% on real-world subsets according to GPT4o-as-judge, and achieving the best average human evaluation across all subsets of AURORA-BENCH.

## Qualitative Examples
![Qualitative Examples](https://raw.githubusercontent.com/yfqiu-nlp/vlm-world-model/main/figures/qualitative-example.png)

## Preparation for the Environment

Step 1: initialize the conda environment
```
conda create -f cwm python=3.10
conda activate cwm
```

Step 2: install the requried packages with the following version,
```
torch==2.6.0
peft==0.15.2
flash-attn==2.7.4
```
and install the transformers from the source
```
cd transformers
pip install -e .
```

Step 3: download the AURORA benchmark from their github repo.

```
git clone https://github.com/McGill-NLP/AURORA
```

## Materials

### Model Outputs on AURORA

We release the model's output for **CWM** and **C-FT** avaliable at [here](https://drive.google.com/drive/folders/1YhBmaQGRJax5_hvhZP6ye8NDcWld8sZi?usp=sharing) for future research in the community. All output images are indexed according to the AURORA official test set.

### Model Checkpoints on Huggingface Hub

We release all LoRA checkpoints avaliable on Huggingface Hub,

| Model Name | Description | Link |
|------------|-------------|------|
| **CWM** | CWM bootstrapped with unlabelled videos. | [yfqiu-nlp/chameleon-world-model-aurora-bootstrap](https://huggingface.co/yfqiu-nlp/chameleon-world-model-aurora-bootstrap) |
| **C-FT** | CWM with AURORA-only SFT. | [yfqiu-nlp/chameleon-world-model-aurora](https://huggingface.co/yfqiu-nlp/chameleon-world-model-aurora) |
| **CDM** | Dynamics Model trained with AURORA's annotation. | [yfqiu-nlp/chameleon-dynamics-model-aurora](https://huggingface.co/yfqiu-nlp/chameleon-dynamics-model-aurora) |

All LoRA checkpoints are trained and expected to equip with the base model [`leloy/Anole-7b-v0.1-hf`](https://huggingface.co/leloy/Anole-7b-v0.1-hf) for inference.

## How to Run

## Evaluation

```
bash scripts/eval.sh
```

## Training

### Data Processing for Training

Checkout to `data-processing` branch for code for tokenizing the trajectories in training set and computing the weights predicted by recognition model.

### SFT for Dynamics Model

```
bash scripts/cdm-train.sh
```

### SFT for CWM

```
bash scripts/cwm-train.sh
```

## Acknowledgment

Part of the code is developed from [leloykun-transformers](https://github.com/leloykun/transformers) and [anole](https://github.com/GAIR-NLP/anole) projects. Thanks for their great works!

vlm-world-model

vlm-world-model 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

vlm-world-model

vlm-world-model 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README