HEAR

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #ai #typescript #llm #go #golang #android #machine-learning #rust #deep-learning #linux

★ 10 Open GitHub ↗

Code for HEAR. Website: https://hear.irmv.top

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center">
 Towards the Vision-Sound-Language-Action Paradigm: 
 The HEAR Framework for Sound-Centric Manipulation
</h1>

Chang Nie, Tianchen Deng, Guangming Wang, Zhe Liu, Hesheng Wang

Shanghai Jiao Tong University, University of Cambridge

<a href="https://hear.irmv.top">Project Page</a>
 |
 <a href="https://hear.irmv.top/#videos">Video Demos</a>
 |
 <a href="https://arxiv.org/abs/2603.16086">arXiv</a>
 |
 <a href="https://arxiv.org/pdf/2603.16086.pdf">Paper PDF</a>
 |
 <a href="https://github.com/IRMVLab/HEAR">Code</a>
 |
 <a href="https://huggingface.co/datasets/biubiu2/OpenX-Sound">Dataset</a>

> HEAR upgrades conventional VLA into a Vision-Sound-Language-Action (VSLA) setting, where a robot must see, hear, remember, and react under delayed, chunked control.

## Overview

Modern VLA policies work well when the key evidence is visually persistent. In sound-centric manipulation, however, the crucial signal may be a short beep, a collision click, the prosody of spoken feedback, or the gradual evolution of a process sound. These cues are often brief, non-repeatable, and easy to miss during open-loop chunk execution.

This work formalizes the **Vision-Sound-Language-Action (VSLA)** paradigm and introduces **HEAR**, an end-to-end framework for sound-causal robot manipulation. HEAR is designed around two core problems:

- **Blind Execution Interval (BEI):** brief audio cues can occur and vanish between policy queries.
- **Temporal grounding:** during long waiting phases with quasi-static vision, the policy still needs to understand how the task is progressing over time.

To address these issues, HEAR combines causal audio memory, multimodal reasoning, future-audio prediction, and smooth action chunk generation.

## What This Work Does

HEAR contributes four pieces together rather than adding audio as a simple extra input:

1. **VSLA paradigm.** We formalize robot control conditioned on multi-view vision, streaming audio, language, and proprioception under delayed decision loops.
2. **HEAR framework.** We introduce a four-part architecture for preserving transient acoustic evidence and grounding long-horizon control in sound.
3. **OpenX-Sound.** We build an audio-augmented pretraining resource from Open X-Embodiment for scalable multisensory pretraining.
4. **HEAR-Bench.** We build the first benchmark explicitly designed for sound-causal manipulation, where visually plausible but acoustically premature actions are counted as failures.

## Framework

HEAR decouples high-frequency auditory sensing from low-frequency decision making:

- **Historizer** keeps a compact causal memory of recent audio packets, so fleeting cues remain available at the next decision.
- **Envisioner** fuses vision, language, proprioception, the current audio window, and the Historizer memory into control-ready multimodal representations.
- **Advancer** predicts near-future audio codes during training, injecting temporal structure into the shared latent space.
- **Realizer** generates smooth action chunks with conditional flow matching, reducing motion jitter and mechanical ego-noise.

<table>
 <tr>
 <td align="center">
 <img src="assets/Historizer.png" alt="Historizer" width="95%"> 
 Historizer 
 Streaming causal audio memory that bridges execution gaps.
 </td>
 </tr>
 <tr>
 <td align="center">
 <img src="assets/Envisioner.png" alt="Envisioner" width="95%"> 
 Envisioner 
 Hierarchical multimodal reasoner for semantic understanding and control features.
 </td>
 </tr>
 <tr>
 <td align="center">
 <img src="assets/Advancer.png" alt="Advancer" width="95%"> 
 Advancer 
 Audio world model for near-future prediction and temporal grounding.
 </td>
 </tr>
</table>

## OpenX-Sound and HEAR-Bench

HEAR is supported by both training data and evaluation infrastructure:

- **OpenX-Sound** augments selected Open X-Embodiment trajectories with synchronized audio for large-scale pretraining. The released dataset is available on Hugging Face at [`biubiu2/OpenX-Sound`](https://huggingface.co/datasets/biubiu2/OpenX-Sound).
- **HEAR-Bench** evaluates seven simulation tasks under strict sound-causal rules across four acoustic cue types: event-triggered alarms, human speech and prosody, continuous process sounds, and physical interaction feedback.

In the paper, OpenX-Sound is manually audited with **98.7% synchronization accuracy within 100 ms tolerance**, and HEAR-Bench is designed so that acting before the required cue is always counted as failure.

## HEAR-Bench Tasks

### Simulation Tasks

<table>
 <tr>
 <td align="center" width="50%">
 <img src="assets/alarmclock.png" alt="Alarm Clock" width="95%"> 
 Alarm Clock 
 Wait for a sustained trigger and press only after ring onset.
 </td>
 <td align="center" width="50%">
 <img src="assets/microwave.png" alt="Microwave" width="95%"> 
 Microwave 
 React to a short ding that may vanish during chunked execution.
 </td>
 </tr>
 <tr>
 <td align="center" width="50%">
 <img src="assets/check_yes.png" alt="Check Yes" width="95%"> 
 Check Yes 
 Use prosody rather than transcript text to decide whether to switch objects.
 </td>
 <td align="center" width="50%">
 <img src="assets/check_materials.png" alt="Check Materials" width="95%"> 
 Check Materials 
 Infer material properties from impact acoustics.
 </td>
 </tr>
 <tr>
 <td align="center" width="50%">
 <img src="assets/pour_water.png" alt="Pour Water" width="95%"> 
 Pour Water 
 Monitor continuous pouring sound to stop at the right target level.
 </td>
 <td align="center" width="50%">
 <img src="assets/boil_water.png" alt="Boil Water" width="95%"> 
 Boil Water 
 Wait through a long static scene and react to the acoustic transition to rolling boil.
 </td>
 </tr>
 <tr>
 <td align="center" width="50%">
 <img src="assets/interrupt.png" alt="Interrupt" width="95%"> 
 Interrupt 
 Safely override an ongoing routine after a spoken reset command.
 </td>
 <td align="center" width="50%"></td>
 </tr>
</table>

### Real-Robot Tasks

<table>
 <tr>
 <td align="center" width="50%">
 <img src="assets/moka_coffee.png" alt="Moka Coffee" width="95%"> 
 Moka Coffee 
 Long-horizon process monitoring from subtle brewing acoustics.
 </td>
 <td align="center" width="50%">
 <img src="assets/answer_phone.png" alt="Answer Phone" width="95%"> 
 Answer Phone 
 Multi-stage task with ringtone detection, speech understanding, and end-of-call recognition.
 </td>
 </tr>
 <tr>
 <td align="center" width="50%">
 <img src="assets/shake_bottle.png" alt="Shake Bottle" width="95%"> 
 Shake Bottle 
 Active acoustic sensing through self-generated shaking motion.
 </td>
 <td align="center" width="50%">
 <img src="assets/real_alarmclock.png" alt="Real Alarm Clock" width="95%"> 
 Real Alarm Clock 
 Robust waiting under real-world reverberation and background noise.
 </td>
 </tr>
</table>

More qualitative demos are available on the [project page](https://hear.irmv.top/#videos).

## Code Release

This repository now contains both the public HEAR training code and the HEAR-Bench benchmark release.

- [`hear/`](hear/) provides the HEAR training and inference codebase built on top of openpi, including the Qwen3-Omni PyTorch training path, dataset conversion tools, and release-ready install instructions.
- [`hear/README.md`](hear/README.md) contains the full setup guide for dependencies, upstream openpi asset compatibility, Hugging Face weight downloads, training, and policy serving.
- HEAR-specific public weights are released on Hugging Face:
  - `biubiu2/HEAR-Qwen3-Omni-30B-A3B-Instruct-Pruned`
  - `biubiu2/HEAR-Qwen3-0.6B`
  - `biubiu2/HEAR-mimi`
- The OpenX-Sound dataset is released on Hugging Face: [`biubiu2/OpenX-Sound`](https://huggingface.co/datasets/biubiu2/OpenX-Sound).
- [`HEAR-Bench/`](HEAR-Bench/) provides the benchmark codebase built on top of RoboTwin 2.0.
- [`HEAR-Bench/README.md`](HEAR-Bench/README.md) contains the full setup guide for installation, asset download, data collection, pi0 data processing, training, and evaluation.
- HEAR-Bench reuses the standard RoboTwin simulator and asset pipeline. Users should follow the RoboTwin-style install flow inside `HEAR-Bench/`, then continue with the HEAR-specific data collection and evaluation steps.
- HEAR-specific audio assets are already included in `HEAR-Bench/assets/audios/`, so no second asset package is required beyond the regular RoboTwin assets downloaded by the setup scripts.

Run training commands from `HEAR/hear` and benchmark commands from `HEAR/HEAR-Bench` after cloning this repository.

## Citation

This paper has been accepted by *The International Journal of Robotics Research (IJRR)*. If you find this project useful, please consider citing:

```bibtex
@article{nie2026hear,
  title={Towards the Vision-Sound-Language-Action Paradigm: The HEAR Framework for Sound-Centric Manipulation},
  author={Nie, Chang and Deng, Tianchen and Wang, Guangming and Liu, Zhe and Wang, Hesheng},
  journal={The International Journal of Robotics Research},
  year={2026},
}
```

HEAR

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README