VAP

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

★ 14 Open GitHub ↗

Official implementation of "Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Prompting" — a training-free method that enables frozen VLAs to manipulate user-specific objects among visually similar distractors.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Bring My Cup! ☕<br>Personalizing Vision-Language-Action Models with Visual Attentive Prompting

[![ICML 2026](https://img.shields.io/badge/ICML-2026-1A73E8.svg)](https://icml.cc/Conferences/2026)
[![arXiv](https://img.shields.io/badge/arXiv-2512.20014-b31b1b.svg)](https://arxiv.org/abs/2512.20014)
[![PDF](https://img.shields.io/badge/PDF-Download-red.svg)](https://arxiv.org/pdf/2512.20014)
[![Project Page](https://img.shields.io/badge/Project-Website-blue)](https://vap-project.github.io)
[![HuggingFace](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Models-yellow)](https://huggingface.co/leesangoh)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

<h3>🎉 Accepted at <a href="https://icml.cc/Conferences/2026">ICML 2026</a> 🎉</h3>

<br>

<p align="center" style="font-size: 1.1rem; margin-top: 20px;">
  <i>"To be truly useful in daily life, robots must discern the subtle details that distinguish <strong>'a cup'</strong> from <strong>'my cup.'</strong>"</i>
</p>

<table align="center" style="border-collapse: collapse; border: none;">
  <tr>
    <td align="center" width="25%" style="border: none; padding: 5px;">
      <img src="./static/videos/real_world_put_my_stuffed_toy.gif" width="100%" style="border-radius: 10px;" alt="Put my stuffed toy into the plastic bowl">
      <br><b>"Put my stuffed toy into the plastic bowl"</b>
    </td>
    <td align="center" width="25%" style="border: none; padding: 5px;">
      <img src="./static/videos/real_world_put_brothers_dog_and_my_ornament.gif" width="100%" style="border-radius: 10px;" alt="Put my brother's dog figurine and my ornament into the plastic bowl">
      <br><b>"Put my brother's dog figurine and my ornament into the plastic bowl"</b>
    </td>
    <td align="center" width="25%" style="border: none; padding: 5px;">
      <img src="./static/videos/real_world_put_my_pouch.gif" width="100%" style="border-radius: 10px;" alt="Put my pouch into the plastic bowl">
      <br><b>"Put my pouch into the plastic bowl"</b>
    </td>
    <td align="center" width="25%" style="border: none; padding: 5px;">
      <img src="./static/videos/real_world_put_my_cat_and_brothers_owl.gif" width="100%" style="border-radius: 10px;" alt="Put my cat figurine and my brother's owl figurine into the plastic bowl">
      <br><b>"Put my cat figurine and my brother's owl figurine into the plastic bowl"</b>
    </td>
  </tr>
  <tr>
    <td align="center" width="25%" style="border: none; padding: 5px;">
      <img src="./static/videos/simpler_bridge_camera.gif" width="100%" style="border-radius: 10px;" alt="Put my camera on towel">
      <br><b>"Put my camera on towel"</b>
    </td>
    <td align="center" width="25%" style="border: none; padding: 5px;">
      <img src="./static/videos/simpler_fractal_pen_holder_matching.gif" width="100%" style="border-radius: 10px;" alt="Pick my pen holder">
      <br><b>"Pick my pen holder"</b>
    </td>
    <td align="center" width="25%" style="border: none; padding: 5px;">
      <img src="./static/videos/vlabench_leather_bag.gif" width="100%" style="border-radius: 10px;" alt="Select my leather bag">
      <br><b>"Select my leather bag"</b>
    </td>
    <td align="center" width="25%" style="border: none; padding: 5px;">
      <img src="./static/videos/simpler_bridge_straw_cup.gif" width="100%" style="border-radius: 10px;" alt="Put my straw cup into the basket">
      <br><b>"Put my straw cup into the basket"</b>
    </td>
  </tr>
</table>

<h3><span style="font-weight:700;">VAP</span> enables frozen VLA models to manipulate user-specific objects among visually similar distractors.</h3>

</div>

---

## 💡 What is VAP?

Existing VLA models are great at understanding generic commands ("pick up the cup"), but they fail when asked to "pick up **my** cup" among other similar cups.

**Visual Attentive Prompting (VAP)** solves this by acting as a pair of "personalized glasses" for the robot.
1.  **See & Remember:** It takes a few reference photos of your object.
2.  **Highlight:** It visually detects and highlights the target object in the robot's view.
3.  **Act:** It guides the frozen VLA model to manipulate the correct object without any expensive training or fine-tuning.

---

## 🚀 Run

Once the environments are set up, you can run the benchmarks immediately.

### 1. Personalized-SIMPLER
Evaluate VAP on SimplerEnv with Bridge/Fractal baselines.
```bash
conda activate simpler
bash ./scripts/run_personalized_simpler_vap.sh
```

### 2. Personalized-VLABench
Evaluate VAP on VLABench tasks.
```bash
conda activate vlabench
bash ./scripts/run_personalized_vlabench_vap.sh
```

### 3. Real-world Scenarios
Deploy VAP as a server for real-world robot experiments.
```bash
conda activate vap_server
bash ./scripts/run_realworld_vap_server.sh
```

---

## 🛠️ Installation

### 1. Clone & Assets
First, clone the repository and download the necessary data assets.

```bash
git clone https://github.com/Leesangoh/VAP.git
cd VAP
git submodule update --init --recursive

# Set Environment Variables
export VAP_HOME=$(pwd)
export HF_HOME=$HOME/.cache/huggingface  # Modify if needed
```

**📥 Download Assets**
Please download the files below and place them in the correct directories:

* **User-provided Object Photos** (Unzip into `${VAP_HOME}/datasets`)
    * [https://drive.google.com/file/d/1GMhmpQJ7ylMcYjjrUtxxq8Kl88V2jFr0/view?usp=drive_link](https://drive.google.com/file/d/1GMhmpQJ7ylMcYjjrUtxxq8Kl88V2jFr0/view?usp=drive_link)
* **Personalized-VLABench Assets** (Unzip into `${VAP_HOME}/src/simulation/VLABench/VLABench/assets`)
    * [https://drive.google.com/file/d/1JLoNanInxlTHmc-blqV4imTBBbylyJeV/view?usp=drive_link](https://drive.google.com/file/d/1JLoNanInxlTHmc-blqV4imTBBbylyJeV/view?usp=drive_link)

### 2. Environment Setup
We use 4 separate conda environments to manage dependencies for different baselines. Click the tabs below to expand the installation commands.

<details>
<summary><b>🐍 A. Setup `openpi_torch` Env (For pi0 server)</b></summary>

```bash
conda create -n openpi_torch python=3.11 -y
conda activate openpi_torch

cd $VAP_HOME/src/simulation/models/openpi_torch
pip install -r requirements.txt
cd packages/openpi-client
pip install -e .
cd ../../
pip install -e .

# Replace transformers modules
cd $VAP_HOME/src/simulation/models/openpi_torch
cp -r ./src/openpi/models_pytorch/transformers_replace/* $(python -c "import transformers; print(transformers.__path__[0])")
conda deactivate
```
</details>

<details>
<summary><b>🤖 B. Setup `simpler` Env (For SIMPLER Simulation)</b></summary>

```bash
conda create -n simpler python=3.10 -y
conda activate simpler

cd $VAP_HOME/src/simulation/SIMPLER/ManiSkill2_real2sim
pip install -e .
cd ../
pip install -e .

# Dependencies
pip install torch tensorflow==2.15.0 pandas matplotlib omegaconf mediapy websockets
pip install flax==0.5 jax==0.4.1 msgpack hydra-core einops transformers==4.56.0 torchvision bitsandbytes

# Fix JAX version
pip uninstall jaxlib -y
pip install "jaxlib==0.4.1" -i https://us-python.pkg.dev/ml-oss-artifacts-published/jax/simple/
pip install numpy==1.24.4

conda deactivate
```
</details>

<details>
<summary><b>🧪 C. Setup `vlabench` Env (For VLABench Simulation)</b></summary>

```bash
conda create -n vlabench python=3.10 -y
conda activate vlabench

cd $VAP_HOME/src/simulation/VLABench
pip install -r requirements.txt
pip install -e .
pip install websockets msgpack torchvision
conda deactivate
```
</details>

<details>
<summary><b>🖥️ D. Setup `vap_server` Env (For Real-world Server)</b></summary>

```bash
conda create -n vap_server python=3.10 -y
conda activate vap_server

pip install torch torchvision numpy msgpack websockets pillow requests
pip install --upgrade transformers accelerate
pip install https://github.com/Dao-AILab/flash-attention/releases/download/v2.8.2/flash_attn-2.8.2+cu12torch2.7cxx11abiTRUE-cp310-cp310-linux_x86_64.whl
conda deactivate
```
</details>

### 3. Checkpoints
Download the required model weights.

<details>
<summary><b>📥 Click to expand Checkpoint Download instructions</b></summary>

**A. Paligemma**
```bash
mkdir -p $HF_HOME
cd $HF_HOME
git clone https://huggingface.co/google/paligemma-3b-pt-224
```

**B. Bridge/Fractal Checkpoints**
```bash
mkdir -p $HF_HOME/open-pi-zero
cd $HF_HOME/open-pi-zero

# Bridge
wget -c -O bridge_beta_step19296_2024-12-26_22-30_42.pt \
"https://huggingface.co/allenzren/open-pi-zero/resolve/main/bridge_beta_step19296_2024-12-26_22-30_42.pt?download=true"

# Fractal
wget -c -O fractal_beta_step29576_2024-12-29_13-10_42.pt \
"https://huggingface.co/allenzren/open-pi-zero/resolve/main/fractal_beta_step29576_2024-12-29_13-10_42.pt?download=true"
```

**C. VLABench Checkpoint**
Download `model.safetensors` from the link below and place it in `$HF_HOME/pi05-vlabench`.
* Link: [Huggingface](https://huggingface.co/leesangoh/pi05-vlabench) or [Google Drive](https://drive.google.com/file/d/1YsbdcRRrITMBUTAWjLYKyiTFZ4QfL754/view?usp=sharing)
</details>

---

## 📝 Citation

If you find this work useful in your research, please cite:

```bibtex
@inproceedings{lee2026bring,
  title={Bring My Cup! Personalizing Vision-Language-Action Models with Visual Attentive Prompting},
  author={Lee, Sangoh and Mo, Sangwoo and Han, Wook-Shin},
  booktitle={Proceedings of the International Conference on Machine Learning (ICML)},
  year={2026}
}
```

VAP

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README