MuSS

KO | EN

gitlite — search

#javascript #python #typescript #llm #go #deep-learning #macos #java #nodejs #kubernetes #vue #pytorch #security #openai

(★ 33)

A Large-Scale Dataset and Cinematic Narrative Benchmark for Multi-Shot Subject-to-Video Generation

MuSS Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h3 align="center">
 A Large-Scale Dataset and Cinematic Narrative Benchmark for 
 Multi-Shot Subject-to-Video Generation
</h3>

Haojie Zhang, Di Wu, Bingyan Liu, Linjie Zhong, Yuancheng Wei, Xingsong Ye, Nanqing Liu, Yaling Liang

<a href="https://arxiv.org/abs/2604.23789">Paper</a> |
 <a href="https://zhang-haojie.github.io/project-pages/muss.html">Project Page</a> |
 <a href="#code">Code</a>

## Overview

MuSS is a large-scale cinematic dataset and benchmark for **multi-shot video generation** and **Subject-to-Video generation**. It targets three limitations that are difficult to expose in single-shot settings:

- lack of authentic cinematic narrative logic;
- conflicts between global captions and local shot-level alignment;
- copy-paste shortcuts in Subject-to-Video models.

MuSS contains two complementary data settings: **Complex Cinematic Narrative**, which focuses on montage, shot transitions, and multi-character storytelling, and **Subject-Centric Narrative**, which focuses on preserving the same subject across disjoint shots and viewpoints.

## News

- **2026-04**: Paper released on arXiv.
- **Initial public release**: data construction code is released first. Dataset files, benchmark implementation, and model checkpoints are not included in this repository.

## Dataset

MuSS is sourced from over **3,000 movies** and contains more than **30,000 professionally captioned multi-shot clips**, totaling over **1,000 hours** of high-quality video content.

The construction pipeline first turns raw cinematic footage into high-quality physical shots with coherent captions, then builds cross-shot S2V pairs by sampling reference subjects from disjoint shot contexts.

### Data Examples

**Complex Cinematic Narrative.** Progressive multi-shot captions are aligned to physical shots, capturing shot transitions, scene changes, and multi-character narrative flow.

**Subject-Centric Narrative.** A reference subject is extracted from a disjoint shot, while the target sequence preserves identity across different viewpoints and contexts.

<details>
<summary>More raw cinematic examples</summary>

</details>

## Benchmark

The **Cinematic Narrative Benchmark** evaluates generated videos under realistic multi-shot storytelling conditions. It combines shot boundary parsing, expert perception models, and LMM-based visual-logic assessment.

| Track | Evaluation Goal | Metrics |
| --- | --- | --- |
| **Track 1: Narrative Effectiveness** | Shot-level alignment, transition precision, scene continuity, and visual logic. | `Txt.Align`, `Trans.Dev`, `Scene.Con`, `Con.Gap`, `Scene.Logic`, `Casting.Logic`, `Act.Logic`, `Spat.Logic` |
| **Track 2: Subject Consistency** | Cross-shot identity preservation, subject grounding, motion strength, and anti-copy-paste behavior. | `Subj.Recall`, `Ref-Sub.Con`, `Inter-Sub.Con`, `Act.Str`, `ACP-Var`, `CP-Rate` |

`ACP-Var` measures pose and structural diversity between the reference image and generated frames, explicitly penalizing rigid 2D reference copying.

## Code

This repository currently releases the data-side code. The benchmark implementation is intentionally not included in this initial repository and will be released separately after further verification.

```text
MuSS/
├── download/        # Raw video acquisition from organized yearly YouTube lists
├── main-pipeline/   # Multi-shot video curation and progressive captioning
├── s2v-pipeline/    # Cross-shot subject extraction and S2V pair construction
├── assets/          # README figures converted from the paper and supplementary material
├── CITATION.cff
└── README.md
```

### Quick Start

```bash
git clone https://github.com/<your-org>/MuSS.git
cd MuSS
python3 download/download.py 2011
```

Module-level guides:

- [Download guide](download/README.md)
- [Multi-shot construction pipeline](main-pipeline/README.md)
- [Cross-shot S2V construction pipeline](s2v-pipeline/README.md)

## Citation

```bibtex
@article{zhang2026muss,
  title   = {MuSS: A Large-Scale Dataset and Cinematic Narrative Benchmark for Multi-Shot Subject-to-Video Generation},
  author  = {Zhang, Haojie and Wu, Di and Liu, Bingyan and Zhong, Linjie and Wei, Yuancheng and Ye, Xingsong and Liu, Nanqing and Liang, Yaling},
  journal = {arXiv preprint arXiv:2604.23789},
  year    = {2026}
}
```

## License

The code and dataset license are being finalized. Please check this section before redistribution or commercial use.

## Acknowledgements

MuSS builds on open research infrastructure for video processing, perception, visual-language reasoning, and generative video evaluation. Please also follow the licenses of any third-party models, datasets, and tools used by the individual pipelines.

MuSS

MuSS Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

MuSS

MuSS Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README