llmblueprint - GitLite

KO | EN

gitlite — search

#typescript #android #machine-learning #rust #deep-learning #linux #docker #nodejs #windows #cli #kubernetes #awesome-list #openai #open-source

llmblueprint

(★ 85)

[ICLR 2024] Official code for the paper "LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts"

File Explorer

Download Latest Version (.zip)

- baseline.cpython-38.pyc
- generation.cpython-38.pyc
- helper_functions.cpython-38.pyc
- prompt_templates.cpython-38.pyc
- shared.cpython-38.pyc
- - paint_by_example-checkpoint.ipynb
- - helper.cpython-38.pyc
  - my_paint_by_example.cpython-38.pyc
- - v1.yaml
- - lr_scheduler.py
  - util.py
- - create_square_gt_for_fid.py
  - inference.py
  - inference_test_bench.py
  - modify_checkpoints.py
  - paint_by_example.ipynb
  - read_bbox.py
- - id_list.npy
- environment.yaml
- helper.py
- inference_test_bench.sh
- LICENSE
- main.py
- my_paint_by_example.py
- paint_by_example.ipynb
- setup.py
- test.sh
- train.sh
- livingroom_1.yaml
- assets.md
- iclr_main_figure_arxiv.png
- iclr_main_figure_revised.png
- INSTALL.md
- intro_image_arxiv.png
- - __init__.cpython-38.pyc
  - attention.cpython-38.pyc
  - attention_processor.cpython-38.pyc
  - models.cpython-38.pyc
  - pipelines.cpython-38.pyc
  - sam.cpython-38.pyc
  - transformer_2d.cpython-38.pyc
  - unet_2d_blocks.cpython-38.pyc
  - unet_2d_condition.cpython-38.pyc
- __init__.py
- attention.py
- attention_processor.py
- modeling_utils.py
- models.py
- pipelines.py
- sam.py
- transformer_2d.py
- unet_2d_blocks.py
- unet_2d_condition.py
- - masks1.png
  - masks2.jpg
  - minidemo.gif
  - model_diagram.png
  - notebook1.png
  - notebook2.png
- - package.json
  - postcss.config.js
  - README.md
  - tailwind.config.js
  - tsconfig.json
- - automatic_mask_generator_example.ipynb
  - onnx_model_example.ipynb
  - predictor_example.ipynb
- - amg.py
  - export_onnx_model.py
- - __init__.py
  - automatic_mask_generator.py
  - build_sam.py
  - predictor.py
- .flake8
- .gitignore
- CODE_OF_CONDUCT.md
- CONTRIBUTING.md
- LICENSE
- linter.sh
- README.md
- setup.cfg
- setup.py
- - __init__.cpython-38.pyc
  - latents.cpython-38.pyc
  - parse.cpython-38.pyc
  - utils.cpython-38.pyc
- __init__.py
- latents.py
- parse.py
- utils.py
baseline.py
generation.py
helper_functions.py
main.py
prompt_templates.py
README.md
requirements.txt
shared.py

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts [ICLR 2024]
Visit our [project page](https://hananshafi.github.io/llm-blueprint/)

[Hanan Gani<sup>1</sup>](https://hananshafi.github.io/), [Shariq Farooq Bhat<sup>2</sup>](https://shariqfarooq123.github.io/), [Muzammal Naseer<sup>1</sup>](https://muzammal-naseer.com/), [Salman Khan<sup>1,3</sup>](https://salman-h-khan.github.io/), [Peter Wonka<sup>2</sup>](https://peterwonka.net/)

<sup>1</sup>MBZUAI      <sup>2</sup>KAUST      <sup>3</sup>Australian National University

[![paper](https://img.shields.io/badge/arXiv-Paper-<COLOR>.svg)](https://arxiv.org/abs/2310.10640)

Official implementation of the paper "LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts".

<hr>

## Contents

1. [Updates](#News)
2. [Highlights](#Highlights)
3. [Main Contributions](#Main-Contributions)
4. [Installation](#Installation)
5. [Run LLMBlueprint](#Run-LLMBlueprint)
6. [Results](#Results)
7. [Citation](#Citation)
8. [Contact](#Contact)
9. [Acknowledgements](#Acknowledgements)

<hr>

## Updates

* Code is released. [Feb 12, 2024]
* Our paper is accepted at **ICLR 2024** [Jan 15, 2024]

## Highlights
![intro-diagram](https://github.com/hananshafi/llmblueprint/blob/main/docs/intro_image_arxiv.png)

> **Abstract:** *Diffusion-based generative models have significantly advanced text-to-image generation but encounter challenges when processing lengthy and intricate text prompts describing complex scenes with multiple objects. While excelling in generating images from short, single-object descriptions, these models often struggle to faithfully capture all the nuanced details within longer and more elaborate textual inputs. In response, we present a novel approach leveraging Large Language Models (LLMs) to extract critical components from text prompts, including bounding box coordinates for foreground objects, detailed textual descriptions for individual objects, and a succinct background context. These components form the foundation of our layout-to-image generation model, which operates in two phases. The initial Global Scene Generation utilizes object layouts and background context to create an initial scene but often falls short in faithfully representing object characteristics as specified in the prompts. To address this limitation, we introduce an Iterative Refinement Scheme that iteratively evaluates and refines box-level content to align them with their textual descriptions, recomposing objects as needed to ensure consistency. Our evaluation on complex prompts featuring multiple objects demonstrates a substantial improvement in recall compared to baseline diffusion models. This is further validated by a user study, underscoring the efficacy of our approach in generating coherent and detailed scenes from intricate textual inputs.*
>
<hr>

## Main Contributions
* **Scene Blueprints:** we present a novel approach leveraging Large Language Models (LLMs)
to extract critical components from text prompts, including bounding box coordinates for foreground objects, detailed textual descriptions for individual objects,
and a succinct background context. Utilizing bounding 
* **Global Scene Generation:** Utilzing the bounding box layout and genralized background prompt, we generate an initial image using Layout-to-Image generator.
* **Iterative Refinement Scheme :** Given the initial image, our proposed refinement mechanism iteratively evaluates and refines the box-level content of each object to align
them with their textual descriptions, recomposing objects as needed to ensure consistency.

## Methodology
![main-figure](https://github.com/hananshafi/llmblueprint/blob/main/docs/iclr_main_figure_revised.png)

## Installation
This codebase is tested on Ubuntu 20.04.2 LTS with python 3.8. Follow the below steps to create environment and install dependencies.

```bash
# Create a conda environment
conda create -n llmblueprint python==3.8

# Activate the environment
conda activate llmblueprint

#Install torch
conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.7 -c pytorch -c nvidia

# Install requirements
pip install -r requirements.txt

# Additionally do this step at the end
python -m spacy download en_core_web_md
```

## Run LLMBlueprint

Download the pretrained weights of composition model from [here](https://github.com/Fantasy-Studio/Paint-by-Example) and provide its path in yaml files placed inside configs folder.

#### Generate
```bash
python main.py --config configs/livingroom_1.yaml
```
The hyperparameters and input arguments can be modified inside yaml files. The generated results will be saved in ./outputs folder.

## Contact
Should you have any questions, please contact at hanan.ghani@mbzuai.ac.ae

## Citation
If you use our work, please consider citing:
```bibtex 
@inproceedings{gani2024llm,
            title={{LLM} Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts},
            author={Hanan Gani and Shariq Farooq Bhat and Muzammal Naseer and Salman Khan and Peter Wonka},
            booktitle={The Twelfth International Conference on Learning Representations},
            year={2024},
            url={https://openreview.net/forum?id=mNYF0IHbRy}
            }
```
## Acknowledgements
Our code is built on the repositories of  [LLM Grounded Diffusion](https://github.com/TonyLianLong/LLM-groundedDiffusion) and [Paint by Example](https://github.com/Fantasy-Studio/Paint-by-Example). We thank them for their open-source implementation and instructions.