KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #ai-agents #dsh-plugin #codex #developer-tools #claude-code #windows #desktop-app #typescript #rust #ai-agent #cli

Awesome-HumanView-VideoUnderstanding

★ 29 Open GitHub ↗

[survey] Watch, Remember, Reason: Human-View Video Understanding with MLLMs

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

[![Awesome](https://cdn.rawgit.com/sindresorhus/awesome/d7305f38d29fed78fa85652e3a63e154dd8e8829/media/badge.svg)](https://github.com/sindresorhus/awesome)
[![PR's Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat)](https://github.com/lxtGH/Awesome-HumanView-VideoUnderstanding/pulls)
 

 <h1 align="center">Watch, Remember, Reason: Human-View Video Understanding with MLLMs</h1>
 
 
 <a href="https://marinero4972.github.io/">Jiahao Meng</a>
 ·
 <a href="https://openreview.net/profile?id=~Tan_Yue2">Yue Tan</a>
 ·
 <a href="https://insomniaaac.github.io/">Qi Xu</a>
 ·
 Kuan Gao
 ·
 Weisong Liu
 ·
 <a href="https://yanwei-li.com/">Yanwei Li</a>
 ·
 Jason Li
 
 <a href="https://ldkong.com/">Lingdong Kong</a>
 ·
 <a href="https://haochen-wang409.github.io/">Haochen Wang</a>
 ·
 Qianyu Zhou
 ·
 <a href="https://zhangzjn.github.io/">Jiangning Zhang</a>
 ·
 <a href="https://sites.google.com/view/guangliangcheng">Guangliang Cheng</a>
 
 <a href="https://www.ai.pku.edu.cn/en/info/1459/2082.htm">Yunhai Tong</a>
 ·
 <a href="http://luqi.info/">Lu Qi</a>
 ·
 <a href="https://faculty.ucmerced.edu/mhyang/">Ming-Hsuan Yang</a>

## Overview

This repository accompanies our survey, which takes a **human-view** perspective on LLM/MLLM-based video understanding by decomposing it into three core cognitive abilities and reviewing how recent methods realize each:

- **Watch** — perceptual grounding of multimodal video streams: fine-grained spatio-temporal grounding, comprehensive captioning, audio-visual (omni-modal) perception, and efficient long-video processing.
- **Remember** — maintaining context over long or streaming inputs: offline memory (agentic vs. non-agent) and streaming memory.
- **Reason** — deriving grounded answers from perceived and retained evidence: text-only reasoning and the emerging *thinking-with-videos* paradigm, each split into agentic and non-agent approaches.

Beyond methods, the survey covers domain-specific subfields (egocentric, sports, instructional, medical, narrative videos), training datasets and evaluation benchmarks across major task types and capability dimensions, and open problems on the path to scalable, memory-aware, evidence-grounded video intelligence.

## Table of Contents

- [1. Watch — How to Watch?](#1-watch--how-to-watch)
  - [1.1 Fine-grained Watching](#11-fine-grained-watching)
  - [1.2 Comprehensive Watching](#12-comprehensive-watching)
  - [1.3 Audio-Visual Watching](#13-audio-visual-watching)
  - [1.4 Efficient Watching](#14-efficient-watching)
- [2. Remember — How to Remember?](#2-remember--how-to-remember)
  - [2.1 Offline Memory — Agentic](#21-offline-memory--agentic)
  - [2.2 Offline Memory — Non-Agent](#22-offline-memory--non-agent)
  - [2.3 Streaming Memory](#23-streaming-memory)
- [3. Reason — How to Reason?](#3-reason--how-to-reason)
  - [3.1 Text-only Reasoning — Agentic](#31-text-only-reasoning--agentic)
  - [3.2 Text-only Reasoning — Non-Agent](#32-text-only-reasoning--non-agent)
  - [3.3 Thinking with Videos — Agentic](#33-thinking-with-videos--agentic)
  - [3.4 Thinking with Videos — Non-Agent](#34-thinking-with-videos--non-agent)
- [4. Subfields](#4-subfields)
  - [4.1 Egocentric Videos](#41-egocentric-videos)
  - [4.2 Sports Videos](#42-sports-videos)
  - [4.3 Instructional Videos](#43-instructional-videos)
  - [4.4 Medical Videos](#44-medical-videos)
  - [4.5 Movie and Narrative Videos](#45-movie-and-narrative-videos)
- [5. Datasets and Benchmarks](#5-datasets-and-benchmarks)
  - [5.1 Training Datasets](#51-training-datasets)
  - [5.2 Evaluation Benchmarks](#52-evaluation-benchmarks)
- [6. Future Directions](#6-future-directions)
- [Citation](#citation)
- [Contributing](#contributing)

---

## 1. Watch — How to Watch?

Watching corresponds to the perceptual stage where models transform raw multimodal inputs into structured representations. We organize methods along four complementary dimensions.

### 1.1 Fine-grained Watching

Precise spatio-temporal grounding: time representation, long-video efficiency, structured decoding, fine-grained perception architectures, and verifiable post-training. Includes both flagship methods (Table 2) and the broader family cited in §3.1.1 (temporal grounding, spatio-temporal grounding, video referring).

**Temporal grounding & VTG-style methods**

**Spatio-temporal grounding & video referring**

### 1.2 Comprehensive Watching

Whole-video, dense, and region-level captioning over visual-token sequences. Includes flagship methods (Table 2) and the broader family cited in §3.1.2.

**Whole-video captioning**

**Dense video captioning**

**Region-level captioning & video referring**

### 1.3 Audio-Visual Watching

Omni-modal perception unifying speech, environmental audio, and music with visual streams under a shared LLM backbone.

### 1.4 Efficient Watching

Reducing redundancy in long videos: frame-level selection, token-level compression / merging, and model-level efficient processing.

---

## 2. Remember — How to Remember?

Memory connects perception with higher-level understanding by retaining salient information over time. We split methods by storage paradigm.

### 2.1 Offline Memory — Agentic

LLMs/VLMs autonomously invoke memory tools through multi-round reasoning to construct and retrieve memory.

### 2.2 Offline Memory — Non-Agent

Deterministic pipelines where memory construction and retrieval are sequential, fixed stages.

### 2.3 Streaming Memory

Processing unbounded streams within fixed memory budgets via causal compression, KV pruning, sink reuse, or hierarchical caches.

## 3. Reason — How to Reason?

Reasoning operates on perceived and retained evidence. We separate models by *what* they reason over (text-only vs. interleaved with visual evidence) and *how* the reasoning is controlled (agentic tool loops vs. non-agent post-training).

### 3.1 Text-only Reasoning — Agentic

Agent loops with tools / memory / planning that orchestrate perception, retrieval, verification, and reflection.

### 3.2 Text-only Reasoning — Non-Agent

CoT-style supervised fine-tuning and RL/DPO post-training without tool-driven control loops.

### 3.3 Thinking with Videos — Agentic

Models that interleave reasoning with explicit visual grounding and re-inspection (o3-style "thinking with images" extended to video).

### 3.4 Thinking with Videos — Non-Agent

Models that natively emit grounded reasoning traces (timestamps, boxes, captions) without external tool calls.

---

## 4. Subfields

Domain-specific scenarios that stress different combinations of perception, memory, and reasoning.

### 4.1 Egocentric Videos

### 4.2 Sports Videos

| Year | Venue | Acronym | Paper | Code / Project |
| :---: | :---: | :---: | :--- | :---: |
| 2025 | ICLR | SPORTU | [SPORTU: A Comprehensive Sports Understanding Benchmark for Multimodal Large Language Models](https://arxiv.org/abs/2410.08474) | [Code](https://github.com/chili-lab/SPORTU) |
| 2025 | CVPR | UniSoccer | [Towards Universal Soccer Video Understanding](https://arxiv.org/abs/2412.01820) | [Code](https://github.com/jyrao/UniSoccer) |
| 2025 | CVPR-W | — | [Domain Adaptation of VLM for Soccer Video Understanding](https://arxiv.org/abs/2505.13860) | - |
| 2025 | arXiv | DeepSport | [DeepSport: A Multimodal Large Language Model for Comprehensive Sports Video Reasoning via Agentic Reinforcement Learning](https://arxiv.org/abs/2511.12908) | - |
| 2025 | ACM MM | FineQuest | [FineQuest: Adaptive Knowledge-Assisted Sports Video Understanding via Agent-of-Thoughts Reasoning](https://arxiv.org/abs/2509.11796) | - |
| 2025 | arXiv | TennisTV | [TennisTV: Do Multimodal Large Language Models Understand Tennis Rallies?](https://arxiv.org/abs/2509.15602) | - |
| 2026 | arXiv | — | [Learning Consistent Temporal Grounding between Related Tasks in Sports Coaching](https://arxiv.org/abs/2603.18453) | - |

### 4.3 Instructional Videos

### 4.4 Medical Videos

### 4.5 Movie and Narrative Videos

---

## 5. Datasets and Benchmarks

### 5.1 Training Datasets

#### 5.1.1 Video QA

Covers both training corpora and widely used QA benchmarks (many pre-MLLM datasets are still used for evaluation).

#### 5.1.2 Video Captioning

Includes classical clip / dense / narration-level corpora, MLLM-era recaptioned corpora, and grounded / omni-modal captioning data.

#### 5.1.3 Video Temporal Grounding

Includes large-scale narrated-video pretraining corpora, MLLM-era instruction tuning corpora, and reasoning-/RL-oriented re-annotated data.

#### 5.1.4 Long Video Memory

### 5.2 Evaluation Benchmarks

Categorized into six capability dimensions.

#### 5.2.1 General Video Understanding

#### 5.2.2 Temporal & Spatial Understanding

#### 5.2.3 Complex Reasoning

#### 5.2.4 Long-Context & Streaming Understanding

#### 5.2.6 Omnimodal Collaboration

---

## 6. Future Directions

Open problems highlighted by the survey:

- **Spatial reasoning in video understanding** — bridging object-level and scene-level spatial understanding for embodied / 3D-aware applications.
- **Multi-video and multi-segment temporal grounding** — set-based retrieval + refinement under verifiable rewards.
- **Hour-scale video understanding with structured memory** — multi-level memory with evidence pointers and learned write/forget policies.
- **Efficient and verifiable video reasoning** — budgeted evidence search jointly optimizing answer correctness, evidence alignment, and compactness.
- **Streaming egocentric video understanding** — stateful, goal-driven streaming memory with proactive timing for embodied agents.

---

## Citation

If you find this survey useful, please consider citing:

```bibtex
@article{meng2026watch,
  title   = {Watch, Remember, Reason: Human-View Video Understanding with MLLMs},
  author  = {Meng, Jiahao and Tan, Yue and Xu, Qi and Gao, Kuan and Liu, Weisong and Li, Yanwei and Li, Jason and Kong, Lingdong and Wang, Haochen and Zhou, Qianyu and Zhang, Jiangning and Cheng, Guangliang and Tong, Yunhai and Qi, Lu and Yang, Minghsuan},
  journal = {arXiv preprint arXiv:2606.07433},
  year    = {2026}
}
```

## Contributing

Contributions are welcome — please open a PR to add a missing paper, dataset, or benchmark. When adding an entry, place it in the most specific subsection that fits the taxonomy and follow the existing column order. Papers are listed roughly in chronological order within each table.

Awesome-HumanView-VideoUnderstanding

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README