UVE

KO | EN

gitlite — search

#react #ai #llm #go #android #docker #nodejs #java #windows #cli #pytorch #security #mcp #open-source

(★ 10)

No description available.

UVE Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# UVE: Are MLLMs Unified Evaluators for AI-Generated Videos?

This repository contains:
- **UVE**: A framework that adopts MLLMs to evaluate any aspect of AI-generated videos.
- **Evaluation on UVE-Bench**: A benchmark that assesses the ability of MLLMs to evaluate AI-generated videos.

## UVE
### Quick Start
To begin with, setup the envionment:
```bash
bash scripts/setup_env.sh
```

<details>
<summary><span id="example_videos"> Example Videos </span></summary>
<div style="display: flex; justify-content: space-between;">
  <div style="text-align: center; width: 45%;">
    <h3>mochi_00002.mp4</h3>
    <video src="./example_videos/mochi_00002.mp4" width="100%"></video>
  </div>
  <div style="text-align: center; width: 45%;">
    <h3>OpenSora1.2_00002.mp4</h3>
    <video src="./example_videos/OpenSora1.2_00002.mp4" width="100%"></video>
  </div>
</div>
</details>

**Evaluate pre-defined aspects:**
```python
from uve import UVE

# Initialize the evaluator
evaluator = UVE(model_name='qwen2-vl-7b', max_num_frames=16)

# Evaluate subject structural correctness of a single video
video_path = 'example_videos/mochi_00002.mp4'
result = evaluator.evaluate(video_path, aspect='structural_correctness')

# Evaluate video-text alignment of a single video
video_path = 'example_videos/mochi_00002.mp4'
result = evaluator.evaluate(video_path, aspect='tv_alignment', video_prompt='a man wearing red hat staring at the camera')

# Compare subject structural correctness of a video pair
video_path1 = 'example_videos/mochi_00002.mp4'
video_path2 = 'example_videos/OpenSora1.2_00002.mp4'
result = evaluator.evaluate([video_path1, video_path2], aspect='structural_correctness', eval_mode='pairwise')

# Compare dynamic degree of a video pair
video_path1 = 'example_videos/mochi_00002.mp4'
video_path2 = 'example_videos/OpenSora1.2_00002.mp4'
result = evaluator.evaluate([video_path1, video_path2], aspect='dynamic_degree', eval_mode='pairwise')
```

**Evaluate customized aspects:**
```python
from uve import UVE

# Initialize the evaluator and customized settings
evaluator = UVE(model_name='qwen2-vl-7b', max_num_frames=16)
custom_prompt = "Is the video containing sexual or violent material?\nPlease directly answer yes or no:"
pos_tokens = ['yes', 'Yes', 'YES']
neg_tokens = ['no', 'No', 'NO']

video_path = 'example_videos/mochi_00002.mp4'
result = evaluator.evaluate(video_path, eval_mode='single_soft_custom', custom_prompt=custom_prompt, pos_tokens=pos_tokens, neg_tokens=neg_tokens)
```

**You can also evaluate using this shell script**
```bash
bash scripts/eval_example.sh
```

### Parameters
- **model_name**: The name of the MLLM model. Currently supported models are: qwen2-vl-2b, qwen2-vl-7b, qwen2-vl-72b, internvl-2.5-2b-mpo, internvl-2.5-4b-mpo, internvl-2.5-8b-mpo, internvl-2.5-26b-mpo, internvl-2.5-38b-mpo, internvl-2.5-78b-mpo, llava-onevision-0.5b, llava-onevision-7b, llava-onevision-72b, llava-video-7b, llava-video-72b, minicpm-v-2.6, gpt4o, videoscore, videoscore-v1.1
- **max_num_frames**: The maximum number of frames to sample from each video.
- **video_path**: The path to the video file. For single video evaluation it is a string. For pairwise video evaluation it is a list of strings.
- **video_prompt** (optional): The text prompt used to generate the video.
- **custom_prompt** (optional): The prompt for customized evaluation.
- **eval_mode**: The evaluation mode. Supported modes are:

- **aspect**: The aspect to evaluate. Pre-defined aspects are:

## Evaluation on UVE-Bench

<p align="center">
  <img src="./asset/overview.png" alt=overview width="800"/>
  <br>
  <i>Overview of UVE-Bench. (a) The distribution of video sources. (b) The distribution of data example over 15 fine-grained AIGV evaluation aspects. (c) The distribution of human preference over the four categories. (d) Data examples illustrating how to evaluate both single video rating and video pair comparison using the human preference annotations.</i>
</p>

### UVE-Bench
UVE-Bench is a benchmark designed to assess the ability of MLLMs to evaluate AI-generated videos. It consists of 1,230 videos and human annotated pairwise preferences for 15 fine-grained AIGV evaluation aspects.

For example:
```python
{
    "video_a": "moviegen_480p/moviegen_480p_00000.mp4",
    "video_b": "mochi/mochi_00000.mp4",
    "prompt": null,
    "preference": "B is better",
    "aspect": "dynamic_degree",
    "subaspects": [
        "dynamic_degree",
        "subject_motion_degree"
    ],
    "dataset": "movie_gen_video_bench"
}
```

### Evaluation

**Video Preparation**    
Download the videos from this [link](https://huggingface.co/datasets/lyx97/UVE-Bench) to the folder `uve_bench_videos/`

**Evaluate MLLMs**  
```bash
# Single video rating
bash scripts/eval_uve_bench_single.sh

# Video pair comparison
bash scripts/eval_uve_bench_pair.sh
```

**Evaluate VBench Metrics**
1. Download the VBench models according to this [link](https://github.com/Vchitect/VBench/tree/master/pretrained).
2. Setup VBench environment: `bash scripts/setup_vbench.sh`
3. Convert UVE-Bench annotations to VBench format: `python3 anno2vbench_info.py`
4. Evaluate VBench metrics: `bash scripts/eval_vbench.sh`

UVE

UVE Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

UVE

UVE Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README