MuSS

KO | EN

Minimalist Open Source Repository Search Engine

gitlite — search

#typescript #ai-agents #deepseek-harness #dsh-plugin #open-source #ai #cli #claude-code #windows #codex #developer-tools #dsh

★ 33 Open GitHub ↗

A Large-Scale Dataset and Cinematic Narrative Benchmark for Multi-Shot Subject-to-Video Generation

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h3 align="center">
 A Large-Scale Dataset and Cinematic Narrative Benchmark for 
 Multi-Shot Subject-to-Video Generation
</h3>

Haojie Zhang, Di Wu, Bingyan Liu, Linjie Zhong, Yuancheng Wei, Xingsong Ye, Nanqing Liu, Yaling Liang

<a href="https://arxiv.org/abs/2604.23789">Paper</a> |
 <a href="https://zhang-haojie.github.io/project-pages/muss.html">Project Page</a> |
 <a href="#code">Code</a>

## Overview

MuSS is a large-scale cinematic dataset and benchmark for **multi-shot video generation** and **Subject-to-Video generation**. It targets three limitations that are difficult to expose in single-shot settings:

- lack of authentic cinematic narrative logic;
- conflicts between global captions and local shot-level alignment;
- copy-paste shortcuts in Subject-to-Video models.

MuSS contains two complementary data settings: **Complex Cinematic Narrative**, which focuses on montage, shot transitions, and multi-character storytelling, and **Subject-Centric Narrative**, which focuses on preserving the same subject across disjoint shots and viewpoints.

## News

- **2026-04**: Paper released on arXiv.
- **Initial public release**: data construction code is released first. Dataset files, benchmark implementation, and model checkpoints are not included in this repository.

## Dataset

MuSS is sourced from over **3,000 movies** and contains more than **30,000 professionally captioned multi-shot clips**, totaling over **1,000 hours** of high-quality video content.

The construction pipeline first turns raw cinematic footage into high-quality physical shots with coherent captions, then builds cross-shot S2V pairs by sampling reference subjects from disjoint shot contexts.

### Data Examples

**Complex Cinematic Narrative.** Progressive multi-shot captions are aligned to physical shots, capturing shot transitions, scene changes, and multi-character narrative flow.

**Subject-Centric Narrative.** A reference subject is extracted from a disjoint shot, while the target sequence preserves identity across different viewpoints and contexts.

<details>
<summary>More raw cinematic examples</summary>

</details>

## Benchmark

The **Cinematic Narrative Benchmark** evaluates generated videos under realistic multi-shot storytelling conditions. It combines shot boundary parsing, expert perception models, and LMM-based visual-logic assessment.

| Track | Evaluation Goal | Metrics |
| --- | --- | --- |
| **Track 1: Narrative Effectiveness** | Shot-level alignment, transition precision, scene continuity, and visual logic. | `Txt.Align`, `Trans.Dev`, `Scene.Con`, `Con.Gap`, `Scene.Logic`, `Casting.Logic`, `Act.Logic`, `Spat.Logic` |
| **Track 2: Subject Consistency** | Cross-shot identity preservation, subject grounding, motion strength, and anti-copy-paste behavior. | `Subj.Recall`, `Ref-Sub.Con`, `Inter-Sub.Con`, `Act.Str`, `ACP-Var`, `CP-Rate` |

`ACP-Var` measures pose and structural diversity between the reference image and generated frames, explicitly penalizing rigid 2D reference copying.

## Code

This repository currently releases the data-side code. The benchmark implementation is intentionally not included in this initial repository and will be released separately after further verification.

```text
MuSS/
├── download/        # Raw video acquisition from organized yearly YouTube lists
├── main-pipeline/   # Multi-shot video curation and progressive captioning
├── s2v-pipeline/    # Cross-shot subject extraction and S2V pair construction
├── assets/          # README figures converted from the paper and supplementary material
├── CITATION.cff
└── README.md
```

### Quick Start

```bash
git clone https://github.com/<your-org>/MuSS.git
cd MuSS
python3 download/download.py 2011
```

Module-level guides:

- [Download guide](download/README.md)
- [Multi-shot construction pipeline](main-pipeline/README.md)
- [Cross-shot S2V construction pipeline](s2v-pipeline/README.md)

## Citation

```bibtex
@article{zhang2026muss,
  title   = {MuSS: A Large-Scale Dataset and Cinematic Narrative Benchmark for Multi-Shot Subject-to-Video Generation},
  author  = {Zhang, Haojie and Wu, Di and Liu, Bingyan and Zhong, Linjie and Wei, Yuancheng and Ye, Xingsong and Liu, Nanqing and Liang, Yaling},
  journal = {arXiv preprint arXiv:2604.23789},
  year    = {2026}
}
```

## License

The code and dataset license are being finalized. Please check this section before redistribution or commercial use.

## Acknowledgements

MuSS builds on open research infrastructure for video processing, perception, visual-language reasoning, and generative video evaluation. Please also follow the licenses of any third-party models, datasets, and tools used by the individual pipelines.

MuSS

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README