KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#typescript #ai-agents #developer-tools #codex #claude-code #cli #react #mcp #open-source #deepseek-harness #dsh-plugin #ai

RACCooN

★ 37 Open GitHub ↗

(EMNLP 2025 Main) RACCooN: A Versatile Instructional Video Editing Framework with Auto-Generated Narratives

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# RACCooN: A Versatile Instructional Video Editing Framework with Auto-Generated Narratives
[![Project Website](https://img.shields.io/badge/Project-Website-blue)](https://raccoon-mllm-gen.github.io/)  [![arXiv](https://img.shields.io/badge/arXiv-2405.18406-b31b1b.svg)](https://arxiv.org/abs/2405.18406)   [![HuggingFace](https://img.shields.io/badge/🤗-HuggingFace%20-cyan.svg)](https://huggingface.co/jaehong31/RACCooN)

### Authors: [Jaehong Yoon*](https://jaehong31.github.io/), [Shoubin Yu*](https://yui010206.github.io/), [Mohit Bansal](https://www.cs.unc.edu/~mbansal/)
### University of North Carolina at Chapel Hill

# Setup

## Install Dependencies

1. (Optional) Creating conda environment

```bash
conda create -n RACCooN python=3.10.13
conda activate RACCooN
```

2. build from source

```bash
pip install -r requirements.txt
```

## Download Models & Data

### Video Data
Our VPLM dataset is based on [ROVI](https://github.com/jianzongwu/Language-Driven-Video-Inpainting) videos, please refer to [ROVI project](https://jianzongwu.github.io/projects/rovi/) page to download raw videos and inpainted videos.

### Pre-trained Models

**Visual Encoder**: we adopt pre-trained [ViT-G](https://arxiv.org/abs/2303.15389) (1B), the codebase downloads the model automatically.

**Video-LLM**: we build our MLLM base on [PG-Video-LLaVA](https://github.com/mbzuai-oryx/Video-LLaVA), please refer to the project homepage to setup the Video-LLM

**Diffusion Model**: we fine-tune our video inpainting model based on [StabelDiffusion2.0-inpainting](https://huggingface.co/stabilityai/stable-diffusion-2-inpainting), please download the model to further finetune the model as described in our paper.

### V2P Fine-tuned Models

### P2V Fine-tuned Models

# Dataset Preparation & Feature Extraction
We test our model on:

+ [VPLM](https://huggingface.co/jaehong31/RACCooN/tree/main/VPLM): we follow [ROVI data](https://huggingface.co/datasets/jianzongwu/rovi).

# Stage 1: Training and Inference (Video-to-Paragraph)
We provide RACCooN training and inference script examples as follows.

## 1) Training

```bash
cd v2p

sh scripts/v2p/finetune/vplm.sh
```

## 2) Inference

```bash
cd v2p

sh scripts/v2p/inference/vlpm.sh
```

# Stage 2: Training and Inference (Paragraph-to-Video)
We provide RACCooN training and inference script examples as follows. 
Our code is buit upon [MGIE](https://github.com/apple/ml-mgie). Please setup envoriment following MGIE instruction.

## 1) Training

```bash
cd p2v

sh train.sh
```

## 2) Inference

we provide jupynote [scripts](https://github.com/jaehong31/RACCooN/blob/main/p2v/evaluate.ipynb) for P2V inference.

### TODO 
- [x] Release ckpts and VPLM dataset
- [ ] Video Feature Extraction Examples
- [x] V2P and P2V training
- [ ] Incorporate Grounding Modules

# Acknowledgments
The code is built upon [PG-Video-LLaVA](https://github.com/mbzuai-oryx/Video-LLaVA), [MGIE](https://github.com/apple/ml-mgie), [GroundingDino](https://github.com/IDEA-Research/Grounded-Segment-Anything), and [LGVI](https://github.com/jianzongwu/Language-Driven-Video-Inpainting).

# Reference
Please cite our paper if you use our models in your works:

```bibtex
@inproceedings{yoon2025raccoon,
    title={RACCooN: A Versatile Instructional Video Editing Framework with Auto-Generated Narratives},
    author={Yoon, Jaehong and Yu, Shoubin and Bansal, Mohit},
    booktitle={Conference on Empirical Methods in Natural Language Processing},
    year={2025},
}

RACCooN

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README