Mojito

KO | EN

gitlite — search

#javascript #python #css #html #java #sql #docker #css3 #automation #security #typescript #golang

★ 62 Open GitHub ↗

Official repo for the paper "Mojito: Motion Trajectory and Intensity Control for Video Generation""

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Mojito: Motion Trajectory and Intensity Control for Video Generation

[Xuehai He](https://sheehan1230.github.io/), [Shuohang Wang](https://scholar.google.com.sg/citations?user=mN-IO6wAAAAJ&hl=zh-CN), [Jianwei Yang](https://jwyang.github.io/), [Xiaoxia Wu](https://xwushirley.github.io/), [Yiping Wang](https://ypwang61.github.io/), [Kuan Wang](https://scholar.google.com/citations?hl=en&user=c1-_-dUAAAAJ&view_op=list_works&sortby=pubdate), [Zheng Zhan](https://zhanzheng8585.github.io/), [Olatunji Ruwase](https://www.microsoft.com/en-us/research/people/olruwase/), [Yelong Shen](https://scholar.google.com/citations?user=S6OFEFEAAAAJ&hl=en), [Xin Eric Wang](https://eric-xw.github.io/)

<a href='https://arxiv.org/abs/2412.08948'><img src='https://img.shields.io/badge/Paper-Arxiv-red'></a> <a href='https://sites.google.com/view/mojito-video'><img src='https://img.shields.io/badge/Project-Page-green'></a> [![](https://img.shields.io/badge/Mojito-Github-purple?logo=github&logoColor=white)](https://github.com/eric-ai-lab/Mojito)
</a>

![Teaser figure](https://raw.githubusercontent.com/UCSB-AI/Mojito/main/figures/teaser.png)

## TODO
- [ ] Gradio Demo
- [ ] Huggingface setup
- [x] Release training-free DMC module
- [ ] Release training code
- [x] Release inference code

## Directional Control

### Environmental Setup

``bash environment.bash
``

### Inference
Download the checkpoint from [Google Drive](https://drive.google.com/file/d/1cydrG4xXchUfGLWAeB-uRf2u7EWgVN_c/view?usp=sharing) and put it under the root path of this folder.

Write your text prompt in prompts.txt.

Run the inference by:
```bash
bash configs/inference/run_text2video.sh
```

To achieve the directional/trajectory guidance, you need to specify the object name and it's spatial locations by (the object name should be in your text prompt):

```bash
--object_phrase # The object name to control the moving direction/trajectory in the generated video.

--bboxes # The location of input bounding boxes
```

You can also specify the number of guidance steps by:

```bash
--guidance_step # The number of steps to performance guidance, the higher the stronger guidance.
```

and guidance scale directly by:
```bash
--loss_scale # The higher scale, the stronger guidance.
```

## News
* `2025.6.10`  Release the inference code of DMC.
* `2025.2.10`  🚀 Release the [project website](https://sites.google.com/view/mojito-video)!

## Citation

```
@article{he2024mojito,
  title   = {Mojito: Motion Trajectory and Intensity Control for Video Generation},
  author  = {Xuehai He and Shuohang Wang and Jianwei Yang and Xiaoxia Wu and Yiping Wang and Kuan Wang and Zheng Zhan and Olatunji Ruwase and Yelong Shen and Xin Eric Wang},
  year    = {2024},
  journal = {arXiv preprint arXiv: 2412.08948}
}
```

Mojito

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README