KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#developer-tools #claude-code #codex #typescript #ai-agents #dsh #dsh-plugin #deepseek-harness #ai-agent #mcp #agent-skills #macos

PUMA

★ 132 Open GitHub ↗

Empowering Unified MLLM with Multi-granular Visual Generation

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## PUMA: Empowering Unified MLLM with Multi-Granular Visual Generation

[Rongyao Fang](https://scholar.google.com/citations?user=FtH3CW4AAAAJ&hl=en)1\*, [Chengqi Duan](https://scholar.google.com/citations?user=r9qb4ZwAAAAJ&hl=zh-CN)2\*, [Kun Wang]()3, [Hao Li](https://scholar.google.com/citations?user=qHqQsY4AAAAJ&hl=zh-CN)1,4, [Hao Tian]()3, [Xingyu Zeng]()3, [Rui Zhao]()3, [Jifeng Dai](https://jifengdai.org/)4,5, [Hongsheng Li](https://www.ee.cuhk.edu.hk/~hsli/)1 :envelope:, [Xihui Liu](https://xh-liu.github.io/)2 :envelope:

1CUHK MMLab, 2HKU MMLab, 3SenseTime, 4Shanghai AI Laboratory, 5Tsinghua University

*Equal contribution, :envelope:Corresponding authors
</div>

:fire: **We will release the code and models soon!**

## :book: Table of Contents
- [Update](#update) 
- [TODO](#todo)
- [Environment Setup](#env)
- [Checkpoint Download](#checkpoint)
- [Multi-granular Visual Decoding](#multi-granular)
- [Abstract](#abstract)
- [Framework](#framework)
- [Multi-granular Semantic Visual Decoding](#decoding)
- [Diverse Text-to-image Generation](#t2i)
- [Image Editing](#image-editing)
- [Image Conditional Generation](#cond_gen)

## <a name="update"></a>:new:Update
- **2025.01.16**: PUMA releases checkpoint and inference code for decoders :fire:.
- **2024.10.18**: PUMA preprint is released on ArXiv :fire:.
- **2024.10.17**: PUMA homepage is now available :fire:.

## <a name="todo"></a>:hourglass: TODO

- [x] Update links to project page :link:
- [x] Release visual encoder and decoders checkpoints :computer:
- [ ] Release MLLM backbone checkpoint :floppy_disk:

## <a name="env"></a>Environment Setup
```
conda create -n puma python==3.8
conda activate puma
pip install -r requirements.txt
```

## <a name="checkpoint"></a>Checkpoint Download
```
# You should first replace the <token> with your huggingface token
python download_ckpt.py
```
For manual downloads, please download checkpoints from [here](https://huggingface.co/LucasFang/PUMA) and put the checkpoints under **./ckpts**.

## <a name="multi-granular"></a>Multi-granular Visual Decoding
```
python infer_detokenizer.py --num_tokens <chosen number from [1, 4, 16, 64, 256]>
```

## <a name="abstract"></a>Abstract

> **PUMA** introduces a unified multimodal large language model framework designed to integrate multi-granular visual generation and understanding. Our model excels in a variety of visual tasks, including diverse text-to-image generation, precise image editing, conditional image generation, and visual understanding. It strikes a balance between generation diversity and controllability, making it a versatile tool for visual tasks.

Read the full paper [here](https://arxiv.org/abs/2410.13861).

## <a name="framework"></a>Framework

- PUMA leverages multi-granular visual representations as unified inputs and outputs for MLLM, allowing it to handle a variety of visual tasks, including text-to-image generation, image editing, inpainting, colorization, conditional generation, and image understanding.

## <a name="decoding"></a>Multi-granular Semantic Visual Decoding

- PUMA's visual decoding process spans five granular image representations (f0 to f4) and corresponding decoders (D0 to D4), which are trained using SDXL. This allows PUMA to achieve precise image reconstruction and semantic-guided generation, supporting both control and diversity in image generation tasks.

## <a name="t2i"></a>Diverse Text-to-image Generation

## <a name="image_editing"></a>Image Editing

## <a name="cond_gen"></a>Image Conditional Generation

## <a name="citation"></a>Citation

If you find PUMA useful in your research, please consider citing us:

```
@article{fang2024puma,
  title     ={PUMA: Empowering Unified MLLM with Multi-Granular Visual Generation},
  author    ={Rongyao Fang, Chengqi Duan, Kun Wang, Hao Li, Hao Tian, Xingyu Zeng, Rui Zhao, Jifeng Dai, Hongsheng Li, Xihui Liu},
  journal   ={arxiv},
  year      ={2024}
}
```

## <a name="license"></a>License

This project is released under the [Apache 2.0 license](LICENSE).

## <a name="contact"></a>Contact

If you have any questions, please feel free to contact rongyaofang@gmail.com.  
[Rongyao Fang](https://scholar.google.com/citations?user=FtH3CW4AAAAJ&hl=en) anticipates graduating in 2025 and is open to both academic and industrial research positions. If you are interested, please feel free to contact.

PUMA

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README