SID-bench

KO | EN

gitlite — search

#python #react #typescript #golang #machine-learning #rust #deep-learning #docker #macos #java #windows #kubernetes #pytorch #open-source

(★ 17)

[ICME 2026] Semantic-Aware Interruption Detection in Spoken Dialogue Systems: Benchmark, Metric, and Model

SID-bench Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Semantic Interruption Detection Benchmark (SID-Bench)

[![arXiv](https://img.shields.io/badge/arXiv-2603.24144-b31b1b.svg)](https://arxiv.org/abs/2603.24144)
[![Hugging Face Dataset](https://img.shields.io/badge/%F0%9F%A4%97-Dataset-blue)](https://huggingface.co/datasets/kxxia/SID-bench)

Official repository for **SID-Bench**, introduced in the paper:  
**"Semantic-Aware Interruption Detection in Spoken Dialogue Systems: Benchmark, Metric, and Model"** (arXiv:2603.24144)  
*Joint work by Qwen Team (Alibaba).*

---

## 📢 Why SID-Bench? (Real-World vs. Synthetic)

Most existing full-duplex benchmarks (e.g., FD-Bench, Full-Duplex-Bench) rely on **synthetic data** (text generated by LLMs and spoken by TTS). While scalable, synthetic speech fails to capture the intricate timing, prosody, hesitations, and emotional nuances of authentic human conversations.

**SID-Bench is the first benchmark for semantic-aware interruption detection built ENTIRELY from real-world human-to-human recorded dialogues.**

### Core Features:
- **100% Real-World Audio**: Extracted from genuine, professionally recorded bilingual (English & Mandarin Chinese) conversations and the Switchboard corpus.
- **Intent-Aware Design**: Differentiates between **genuine interruptions** (user wants to take the floor) and **non-interruptive backchannels** (e.g., "uh-huh", "right", "I see").
- **Robustness Testing**: Includes 500 environmental noise and silence clips to stress-test models against non-semantic acoustic triggers.

---

## Dataset Statistics

| Split | Language | Total Samples | With Interruption | Without Interruption |
|-------|----------|:-------------:|:-----------------:|:--------------------:|
| `en_test_lines.jsonl` | English | 1,600 | 1,100 (68.8%) | 500 (31.3%) |
| `zh_test_lines.jsonl` | Chinese | 1,600 | 1,100 (68.8%) | 500 (31.3%) |
| `silence_noise_test.jsonl` | — | 500 | 0 (0%) | 500 (100%) |
| **Total** | | **3,700** | **2,200** | **1,500** |

---

## Directory Structure

```
Semantic_Interaption/
├── README.md
├── test.ipynb                        # Evaluation notebook (calculates FIR, IRL, APT)
└── test_wavs/
    ├── en_test_lines.jsonl           # English annotations (1,600)
    ├── zh_test_lines.jsonl           # Chinese annotations (1,600)
    ├── silence_noise_test.jsonl      # Silence / noise annotations (500)
    ├── en/                           # English WAV files (1,600)
    ├── zh/                           # Chinese WAV files (1,600)
    └── silence_or_noise/             # Silence / noise WAV files (500)
```

---

## Annotation Format

### Speech splits (`en_test_lines.jsonl`, `zh_test_lines.jsonl`)
```json
{
  "audio":           "MDT_ASR_AI106_A0016_S0005_0_G3265_0318890-0324670.wav",
  "total_nonbreak":  false,
  "duration":        5.78,
  "break_time":      0.125,
  "text_with_break": "<break> Experiencing different cultures, learning new languages, ..."
}
```

### Silence / noise split (`silence_noise_test.jsonl`)
```json
{
  "audio":           "wind_wind_heavy_clean_stdy_air.wav",
  "total_nonbreak":  true,
  "duration":        152.52,
  "break_time":      152.52,
  "text_with_break": null
}
```

---

## Evaluation Metrics

To resolve the speed-robustness trade-off, we adopt two primary metrics and propose one unified metric (**APT**). Reference implementation is in `test.ipynb`.

### 1. FIR — False Interruption Rate (Robustness)
The fraction of non-interruption samples where the model incorrectly predicts an interruption (lower is better).
$$FIR = \frac{|False\ Positives|}{|Total\ Nonbreak\ Samples|}$$

### 2. IRL — Interruption Response Latency (Speed)
The average absolute timing error (in seconds) over true-positive detections (lower is better).
$$IRL = \text{mean}(|T_{\text{model stop}} - T_{\text{break}}|) \quad \text{[over True Positives]}$$

### 3. APT — Average Penalty Time (The Composite Metric)
Proposed in our paper, **APT** assigns a "time cost" to the user for both false alarms and delayed responses:

| Outcome | Condition | Penalty Time |
| :--- | :--- | :--- |
| **True Positive** | Correctly stops | $IRL$ (unavoidable latency) |
| **False Positive** | Catastrophic false alarm | $Duration$ of the planned turn (derails conversation) |
| **False Negative** | Fails to stop | $Duration - T_{\text{break}}$ (forces user to listen to unwanted speech) |
| **True Negative**| Correctly ignores backchannel | $0$ (ideal behavior) |

$$APT = \frac{1}{N} \sum_{i=1}^{N} \text{PenaltyTime}_i \quad \text{(Lower is better)}$$

---

## Main Experimental Results

We evaluated our proposed **SID-model** (an Audio-Transformer + Qwen-0.6B architecture trained using a randomized temporal cropping paradigm) against mainstream full-duplex baselines.

| Model | EN APT ↓ | ZH APT ↓ | Noise/Silence APT ↓ | **Average APT ↓** | **Average FIR ↓** | **Average IRL (s) ↓** |
| :--- | :---: | :---: | :---: | :---: | :---: | :---: |
| **FSMN-VAD** | 3.328 | 3.563 | 4.148 | **3.627** | 0.840 | **0.000** |
| **Freeze-Omni** (Silero VAD) | 2.429 | 1.977 | 1.908 | **2.129** | 0.532 | 0.173 |
| **FireRedChat** (pVAD) | 2.997 | 2.311 | 2.502 | **2.627** | 0.476 | 1.045 |
| **Moshi** (End-to-End) | 1.749 | - | 7.810 | **3.192** | 0.118 | 2.517 |
| **SID-model (Ours)** | **0.921** | **0.656** | **0.315** | **0.711** | **0.124** | 0.389 |

> **Key Finding**: Pure VAD-based systems are "trigger-happy" (high FIR), while fully end-to-end models like Moshi are too slow (high IRL). **Our proposed SID-model achieves a nearly 3-fold reduction in APT compared to the best baseline**, successfully balancing speed and robustness.

---

## Download Dataset

The full dataset (audio + annotations) is hosted on Hugging Face:  
👉 **[kxxia/SID-bench](https://huggingface.co/datasets/kxxia/SID-bench)**

```bash
# Option 1: huggingface_hub CLI
huggingface-cli download kxxia/SID-bench --repo-type dataset --local-dir ./SID-bench

# Option 2: Python snapshot download
from huggingface_hub import snapshot_download
snapshot_download(repo_id="kxxia/SID-bench", repo_type="dataset", local_dir="./SID-bench")
```

---

## Citation

If you find SID-Bench, the APT metric, or our methodology useful, please cite our paper:

```bibtex
@misc{semantic_interruption_benchmark,
      title={Semantic-Aware Interruption Detection in Spoken Dialogue Systems: Benchmark, Metric, and Model}, 
      author={Kangxiang Xia and Bingshen Mu and Xian Shi and Jin Xu and Lei Xie},
      year={2026},
      eprint={2603.24144},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2603.24144}, 
}
```

SID-bench

SID-bench Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

SID-bench

SID-bench Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README