GoViG

KO | EN

gitlite — search

#python #react #go #rust #deep-learning #nodejs #macos #windows #cli #vue #awesome-list #security #mcp #open-source

(★ 28)

Official implementation of paper "GoViG: Goal-Conditioned Visual Navigation Instruction Generation"

GoViG 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center">🧭 GoViG: Goal-Conditioned Visual Navigation Instruction Generation</h1>
 <a href=""></a>
 <a>Fengyi Wu1,*,</a>
 <a>Yifei Dong1,*,</a>
 <a>Zhi-Qi Cheng1,†,</a>
 <a>Yilong Dai1,</a>
 <a>Guangyu Chen1,</a>
 <a>Hang Wang2,</a>
 <a>Qi Dai3,</a>
 <a>Alexander G Hauptmann4</a>
 
 1UW, 2PolyU, 3Microsoft Research, 4CMU 
 
 

 <a href="https://arxiv.org/abs/2508.09547" target="_blank">
 <img src="https://img.shields.io/badge/ArXiv-2508.09547-red">
 </a>
 <a href="https://github.com/F1y1113/GoViG" target="_blank">
 <img src="https://img.shields.io/badge/Project-GoViG-blue">
 </a>
 <a href="https://drive.google.com/file/d/17823NqgVHJ-xfx5bFTcm58g6j7D19Hqq/view?usp=sharing" target="_blank">
 <img src="https://img.shields.io/badge/Googledrive-dataset-purple">
 </a>
<a href="https://github.com/F1y1113/GoViG" target="_blank">
 <img src="https://img.shields.io/badge/License-MIT-green">
</a>

**GoViG** introduces a new task in embodied AI: generating navigation instructions directly from egocentric visual observations of the initial and goal states. Unlike previous methods that rely on semantic maps or structured annotations, GoViG operates purely on egocentric visual input—making it highly adaptable to unseen and unstructured environments.

## 🔍 Overview

 <img src="assists/method_overview.png" alt="task" width="1000"/>

GoViG decomposes the instruction generation task into two interconnected subtasks:

- **Navigation Visualization**  
  Predicts intermediate visual states that bridge the initial and goal views.

- **Instruction Generation with Visual Cues**  
  Synthesizes linguistically coherent and spatially grounded instructions based on both observed and anticipated visuals.

These components are unified within an autoregressive MLLM, trained with tailored objectives to ensure spatial accuracy and linguistic clarity.

## 🧠 Reasoning Strategies

Inspired by human navigation behavior, GoViG supports two multimodal reasoning paradigms:

- **One-Pass Reasoning**: Generates instructions in a single forward pass.
- **Interleaved Reasoning**: Alternates between visual prediction and language generation for incremental planning.

## 📦 Dataset: R2R-Goal

To evaluate GoViG, we introduce **R2R-Goal**, a dataset combining synthetic and real-world trajectories.

## Quick Start

```bash
conda create -n GoViG python=3.10
conda activate GoViG
pip install torch==2.4.0
pip install -r requirements.txt --user
```

### Data

We release a partial dataset for the purpose of debugging and demonstrating the data format, you can find them in [data_samples](data_samples/).
And you can access the full dataset [here](https://drive.google.com/file/d/17823NqgVHJ-xfx5bFTcm58g6j7D19Hqq/view?usp=sharing)

```bash
unzip R2R_Goal.zip
```

### Training

```bash
bash train.sh
```

### Evaluation

```bash
bash eval.sh
```

you can find detailed metrics calculation in [taskeval_vis.py](taskeval_vis.py).

## Acknowledgement

We would like to thank [ANOLE](https://arxiv.org/abs/2407.06135) and [MVOT](https://arxiv.org/abs/2501.07542) for their publicly available codebase, which we referenced during the implementation of Anole training.

## 🧭 GoViG Gallery

<table style="table-layout: fixed; width:2000px;">
 <thead>
 <tr>
 <th style="width:180px;">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Initial View&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th>
 <th style="width:180px;">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Goal View&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th>
 <th style="width:180px;">Trajectory (1P)</th>
 <th style="width:180px;">Instructions (1P)</th>
 <th style="width:180px;">Trajectory (Int)</th>
 <th style="width:180px;">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Instructions (Int)&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td><img src="assists/seen/example1/onepass/initial_obs_0.png" style="width:180px;"></td>
 <td><img src="assists/seen/example1/onepass/goal_obs.png" style="width:180px;"></td>
 <td><img src="assists/seen/example1/onepass/traj1.gif" style="width:180px;"></td>
 <td>Stop in the doorway.</td>
 <td><img src="assists/seen/example1/interleaved/traj1.gif" style="width:180px;"></td>
 <td>Stop in front of the last door on your right. Then take a slight left turn to go towards the bathroom. After you leave the kitchen and go through the double doors, keep going and go into the living room. Turn left at the first door past the oven and continue down the hallway. Go into the powder room that is straight ahead. Walk past the bathroom door.</td>
 </tr>
 <tr>
 <td><img src="assists/seen/example2/onepass/initial_obs_0.png" style="width:180px;"></td>
 <td><img src="assists/seen/example2/onepass/goal_obs.png" style="width:180px;"></td>
 <td><img src="assists/seen/example2/onepass/traj1.gif" style="width:180px;"></td>
 <td>Walk into the bedroom.</td>
 <td><img src="assists/seen/example2/interleaved/traj1.gif" style="width:180px;"></td>
 <td>Walk out of the bedroom using the door on your right. Walk out of bedroom and turn right. Leave the bedroom. Turn to your right and go outside. Exit the room. Exit bedroom through doorway on the right.</td>
 </tr>
 <tr>
 <td><img src="assists/seen/example3/onepass/initial_obs_0.png" style="width:180px;"></td>
 <td><img src="assists/seen/example3/onepass/goal_obs.png" style="width:180px;"></td>
 <td><img src="assists/seen/example3/onepass/traj1.gif" style="width:180px;"></td>
 <td>Across the kitchen.</td>
 <td><img src="assists/seen/example3/interleaved/traj1.gif" style="width:180px;"></td>
 <td>Exit the kitchen. Turn right at the counter. Walk past kitchen island. Turn past the sink, and in front of the oven to your left. Make a left immediately through the kitchenette, then turn right into the hallway. Walk past the sink.</td>
 </tr>
 <tr>
 <td><img src="assists/seen/example4/onepass/initial_obs_0.png" style="width:180px;"></td>
 <td><img src="assists/seen/example4/onepass/goal_obs.png" style="width:180px;"></td>
 <td><img src="assists/seen/example4/onepass/traj1.gif" style="width:180px;"></td>
 <td>Go through the door.</td>
 <td><img src="assists/seen/example4/interleaved/traj1.gif" style="width:180px;"></td>
 <td>Straight through the bedroom with the lamp. Turn left and wait in the doorway. Stop in the bedroom doorway. Then turn right and wait in bedroom at the end of the hall. Stop in the doorway. Turn slight left, continue straight. Turn slight left, stop at bed.</td>
 </tr>
 <tr>
 <td><img src="assists/unseen/example1/onepass/initial_obs_0.png" style="width:180px;"></td>
 <td><img src="assists/unseen/example1/onepass/goal_obs.png" style="width:180px;"></td>
 <td><img src="assists/unseen/example1/onepass/traj1.gif" style="width:180px;"></td>
 <td>Walk out of the kitchen.</td>
 <td><img src="assists/unseen/example1/interleaved/traj1.gif" style="width:180px;"></td>
 <td>Walk through the kitchen stop at the oven. Continue walking straight down the kitchen. Turn left, walk down the kitchen hallway. Turn left and enter kitchen. Walk and stop right before washing area. Turn right and continue down the hall until you get to a refrigerator.</td>
 </tr>
 <tr>
 <td><img src="assists/unseen/example2/onepass/initial_obs_0.png" style="width:180px;"></td>
 <td><img src="assists/unseen/example2/onepass/goal_obs.png" style="width:180px;"></td>
 <td><img src="assists/unseen/example2/onepass/traj1.gif" style="width:180px;"></td>
 <td>Walk past the room on the left.</td>
 <td><img src="assists/unseen/example2/interleaved/traj1.gif" style="width:180px;"></td>
 <td>Walk past the door directly across from you. Continue straight and continue through a second set of double doors. Pass the wall on the right. Turn left and enter kitchen. Go down the hall into the office on the left. Walk to the end of the hall and through the open door.</td>
 </tr>
 <tr>
 <td><img src="assists/unseen/example3/onepass/initial_obs_0.png" style="width:180px;"></td>
 <td><img src="assists/unseen/example3/onepass/goal_obs.png" style="width:180px;"></td>
 <td><img src="assists/unseen/example3/onepass/traj1.gif" style="width:180px;"></td>
 <td>Walk up stairs, turn right, continue up stairs</td>
 <td><img src="assists/unseen/example3/interleaved/traj1.gif" style="width:180px;"></td>
 <td>Walk up stairs. Go up the stairs. Walk straight ahead passed the stairs. Go up the stairs. Go up three steps then wait at the top. Go all of the way up the stairs.</td>
 </tr>
 <tr>
 <td><img src="assists/unseen/example4/onepass/initial_obs_0.png" style="width:180px;"></td>
 <td><img src="assists/unseen/example4/onepass/goal_obs.png" style="width:180px;"></td>
 <td><img src="assists/unseen/example4/onepass/traj1.gif" style="width:180px;"></td>
 <td>Walk past the room on the left.</td>
 <td><img src="assists/unseen/example4/interleaved/traj1.gif" style="width:180px;"></td>
 <td>Stop in entryway of house. Stop at sliding barn door. Wait near the patio. Turn to the front row of couches is showing and walk over to the patio. Wait in the doorway to the patio. Walk straight besides the wooden tables. Stop when you reach the sliding glass doors.</td>
 </tr>
 </tbody>
 </table>
</div>

More examples of GoViG results on the **Real-world** Subset of our **R2R-Goal** dataset.

## 🌟 Citation

If you find this repository or our paper useful, please consider **starring** this repository and **citing** our paper:
```bibtex
@misc{wu2026goviggoalconditionedvisualnavigation,
      title={GoViG: Goal-Conditioned Visual Navigation Instruction Generation via Multimodal Reasoning}, 
      author={Fengyi Wu and Yifei Dong and Yilong Dai and Guangyu Chen and Qifeng Wu and Huiting Huang and Hang Wang and Qi Dai and Alexander G. Hauptmann and Zhi-Qi Cheng},
      year={2026},
      eprint={2508.09547},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2508.09547}, 
}
```

GoViG

GoViG 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

GoViG

GoViG 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README