VideoTG-R1

KO | EN

gitlite — search

#javascript #hacktoberfest #android #machine-learning #deep-learning #linux #nodejs #cli #kubernetes #vue #awesome-list #mcp #ios #open-source

(★ 10)

VideoTG-R1: Boosting Video Temporal Grounding via Curriculum Reinforcement Learning on Reflected Boundary Annotations

VideoTG-R1 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h2 align="center">VideoTG-R1: Boosting Video Temporal Grounding via
Curriculum Reinforcement Learning on Reflected Boundary Annotations</h2>

Lu Dong1,3, Haiyu Zhang2,3, Han Lin4,3, Ziang Yan5,3, Xiangyu Zeng6,3, Hongjie Zhang3, Yifei Huang3, Yi Wang3, Zhen-Hua Ling1, Limin Wang6,3, Yali Wang7,3&dagger;
 
 1University of Science and Technology of China &nbsp;&nbsp;
 2Beihang University &nbsp;&nbsp;
 3Shanghai Artificial Intelligence Laboratory 
 4Shanghai Jiao Tong University &nbsp;&nbsp;
 5Zhejiang University &nbsp;&nbsp;
 6Nanjing University &nbsp;&nbsp;
 7Chinese Academy of Sciences

&dagger; Corresponding author

![xxxxxxxx](https://raw.githubusercontent.com/ldong1111/VideoTG-R1/master/figs/main.png)

> **TL;DR:** Data-Efficient Video Temporal Grounding.

**VideoTG-R1**  is a multi-agent system for data-efficient video temporal grounding. It contains three modules: 1) Boundary Reflection Agent filters the training data by identifying and discarding partially annotated samples; 2) Difficulty Estimation Agent estimates the difficulty of each sample via zero-shot evaluation; and 3) Curriculum RL strategy dynamically masks the videos of hard-to-ground samples according to the training steps, to ease the training difficulty of hard-to-ground samples. VideoTG-R1 achieves state-of-the-art performance on Charades-STA and ActivityNet-Captions. Moreover, with only 10\% of the training samples and 21\% of the computational budget, VideoTG-R1 outperforms full-data counterparts under both group relative policy optimization (GRPO) and supervised fine-tuning (SFT).

![xx](https://raw.githubusercontent.com/ldong1111/VideoTG-R1/master/figs/BRA.png)

![xx](https://raw.githubusercontent.com/ldong1111/VideoTG-R1/master/figs/curriculum_RL.png)

## Code Environment Preparation

pip install requirements.txt

## Dataset Preparation

All training and evaluation datasets can be download in [VideoMind's huggingface](https://huggingface.co/datasets/yeliudev/VideoMind-Dataset/tree/main).

## Evaluation
1. video grounding

```
cd Eval

your_ckpt=xxx
dataset_name=charades # or anet

bash video_grounding.sh ${dataset_name}
```
2. grounded qa
```
cd Eval

your_ckpt=xxx
dataset_name=rextime # or nextgqa

bash grounded_qa.sh ${dataset_name}
```

## Fast Training
1. You should download the **intermediate_results**  in the huggingface first.
```
cd ./videotg_r1

bash grpo_train.sh
```

## Full Training
1. **boundary reflection agent**
    1. evaluate each dataset from 'qvhighlights,didemo,tacos,queryd,hirest_grounding,hirest_step,cosmo_cap,internvid_vtime'
    2. you can run the code with single/multiple gpus.
```
cd ./BRA

dataset_name=cosmo_cap  
each_fold_size=22000
fold_index=0

bash bra_test.sh ${dataset_name} ${each_fold_size} ${fold_index}

```
2. **difficulty estimation agent**
   1. evaluate each dataset from 'qvhighlights,didemo,tacos,queryd,hirest_grounding,hirest_step,cosmo_cap,internvid_vtime'
    2. you can run the code with single/multiple gpus.
```

cd ./DEA

dataset_name=cosmo_cap  
each_fold_size=22000
fold_index=0

bash bra_test.sh ${dataset_name} ${each_fold_size} ${fold_index}

```

3. **curriculum grpo**

```

cd ./videotg_r1

bash grpo_train.sh

```

## Acknowledgement
VideoTG-R1 is built with reference to the following projects: [VideoMind](https://github.com/yeliudev/VideoMind) and [VideoChat-R1](https://github.com/OpenGVLab/VideoChat-R1). Thanks for their work!

VideoTG-R1

VideoTG-R1 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

VideoTG-R1

VideoTG-R1 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README