imagebart - GitLite

KO | EN

gitlite — search

#hacktoberfest #typescript #go #machine-learning #linux #macos #nodejs #awesome #cli #vue #pytorch #security #mcp #open-source

imagebart

(★ 126)

ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis

파일 탐색기

최종 버전 다운로드 (.zip)

- foxchain.png
- image_editing.png
- imagebart_poster.pdf
- modelfigure.png
- sample-001.jpg
- - ffhq_4-scales_joint-training.yaml
- - lsun-beds-scale1.yaml
  - lsun-beds-scale2.yaml
  - lsun-beds-scale3.yaml
- - lsun-cats-scale1.yaml
  - lsun-cats-scale2.yaml
  - lsun-cats-scale3.yaml
- - lsun-churches-scale1.yaml
  - lsun-churches-scale2.yaml
  - lsun-churches-scale3.yaml
- - bedroom_indices.npy
  - cat_indices.npy
  - church_indices.npy
  - ffhq_indices.npy
  - imagenet_indices.npy
- DejaVuSans.ttf
- ffhq_schedule_vs_metric.p
- ffhqtrain.txt
- ffhqvalidation.txt
- imagenet_ids2labels.yaml
- in_schedule_vs_metric.p
- - __init__.py
  - base.py
  - lsun.py
- - __init__.py
  - diffusion.py
  - vqgan.py
- - __init__.py
  - betas.py
  - ema.py
- __init__.py
- lr_scheduler.py
- util.py
- inpaint_imagebart.py
- sample_imagebart.py
environment.yaml
main.py
README.md
setup.py

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/CompVis/imagebart

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd imagebart

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install .

PyPI에 배포된 패키지를 바로 설치합니다. 소스 클론이 필요 없습니다.

python main.py

파이썬 스크립트(또는 모듈)를 실행합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# ImageBART
#### [NeurIPS 2021](https://nips.cc/)

![teaser](https://raw.githubusercontent.com/CompVis/imagebart/master/assets/modelfigure.png)
<br/>
[Patrick Esser](https://github.com/pesser)\*,
[Robin Rombach](https://github.com/rromb)\*,
[Andreas Blattmann](https://github.com/ablattmann)\*,
[Björn Ommer](https://ommer-lab.com/)<br/>
\* equal contribution

[arXiv](https://arxiv.org/abs/2108.08827) | [BibTeX](#bibtex) | [Poster](assets/imagebart_poster.pdf)

## Requirements
A suitable [conda](https://conda.io/) environment named `imagebart` can be created
and activated with:

```
conda env create -f environment.yaml
conda activate imagebart
```

## Get the Models

We provide pretrained weights and hyperparameters for models trained on the following datasets:

* FFHQ: 
    * [4 scales, geometric noise schedule](https://ommer-lab.com/files/ffhq_4_scales_geometric.zip): `wget -c https://ommer-lab.com/files/ffhq_4_scales_geometric.zip`
    * [2 scales, custom noise schedule](https://ommer-lab.com/files/ffhq_2_scales_custom.zip): `wget -c https://ommer-lab.com/files/ffhq_2_scales_custom.zip`
* LSUN, 3 scales, custom noise schedules:
    * [Churches](https://ommer-lab.com/files/churches_3_scales.zip): `wget -c https://ommer-lab.com/files/churches_3_scales.zip`
    * [Bedrooms](https://ommer-lab.com/files/bedrooms_3_scales.zip): `wget -c https://ommer-lab.com/files/bedrooms_3_scales.zip`
    * [Cats](https://ommer-lab.com/files/cats_3_scales.zip): `wget -c https://ommer-lab.com/files/cats_3_scales.zip`
* Class-conditional ImageNet:
    * [5 scales, custom noise schedule](https://ommer-lab.com/files/cin_5_scales_custom.zip): `wget -c https://ommer-lab.com/files/cin_5_scales_custom.zip`
    * [4 scales, geometric noise schedule](https://ommer-lab.com/files/cin_4_scales_geometric.zip): `wget -c https://ommer-lab.com/files/cin_4_scales_geometric.zip`

Download the respective files and extract their contents to a directory `./models/`.

Moreover, we provide all the required VQGANs as a .zip at [https://ommer-lab.com/files/vqgan.zip](https://ommer-lab.com/files/vqgan.zip), 
which contents have to be extracted to `./vqgan/`.

## Get the Data
Running the training configs or the [inpainting script](scripts/inpaint_imagebart.py) requires 
a dataset available locally. For ImageNet and FFHQ, see this repo's parent directory [taming-transformers](https://github.com/CompVis/taming-transformers).
The LSUN datasets can be conveniently downloaded via the script available [here](https://github.com/fyu/lsun).
We performed a custom split into training and validation images, and provide the corresponding filenames
at [https://ommer-lab.com/files/lsun.zip](https://ommer-lab.com/files/lsun.zip). 
After downloading, extract them to `./data/lsun`. The beds/cats/churches subsets should
also be placed/symlinked at `./data/lsun/bedrooms`/`./data/lsun/cats`/`./data/lsun/churches`, respectively.

## Inference

### Unconditional Sampling
We provide a script for sampling from unconditional models trained on the LSUN-{bedrooms,bedrooms,cats}- and FFHQ-datasets.

#### FFHQ

On the FFHQ dataset, we provide two distinct pretrained models, one with a chain of length 4 and a geometric noise schedule as proposed by Sohl-Dickstein et al. [[1]](##References) , and another one with a chain of length 2 and a custom schedule.
These models can be started with
```shell script
CUDA_VISIBLE_DEVICES=<gpu_id> streamlit run scripts/sample_imagebart.py configs/sampling/ffhq/<config>
```

#### LSUN
For the models trained on the LSUN-datasets, use 
```shell script
CUDA_VISIBLE_DEVICES=<gpu_id> streamlit run scripts/sample_imagebart.py configs/sampling/lsun/<config>
```

### Class Conditional Sampling on ImageNet

To sample from class-conditional ImageNet models, use
```shell script
CUDA_VISIBLE_DEVICES=<gpu_id> streamlit run scripts/sample_imagebart.py configs/sampling/imagenet/<config>
```

### Image Editing with Unconditional Models

We also provide a script for image editing with our unconditional models. For our FFHQ-model with geometric schedule this can be started with
```shell script
CUDA_VISIBLE_DEVICES=<gpu_id> streamlit run scripts/inpaint_imagebart.py configs/sampling/ffhq/ffhq_4scales_geometric.yaml
```
resulting in samples similar to the following.
![teaser](https://raw.githubusercontent.com/CompVis/imagebart/master/assets/image_editing.png)

## Training
In general, there are two options for training the autoregressive transition probabilities of the 
reverse Markov chain: (i) train them jointly, taking into account a weighting of the 
individual scale contributions, or (ii) train them independently, which means that each 
training process optimizes a single transition and the scales must be stacked after training. 
We conduct most of our experiments using the latter option, but provide configurations for both cases.

### Training Scales Independently
For training scales independently, each transition requires a seperate optimization process, which can 
started via

```
CUDA_VISIBLE_DEVICES=<gpu_id> python main.py --base configs/<data>/<config>.yaml -t --gpus 0, 
```

We provide training configs for a four scale training of FFHQ using a geometric schedule, 
a four scale geometric training on ImageNet and various three-scale experiments on LSUN.
See also the overview of our [pretrained models](#get-the-models).

### Training Scales Jointly

For completeness, we also provide a config to run a joint training with 4 scales on FFHQ.
Training can be started by running

```
CUDA_VISIBLE_DEVICES=<gpu_id> python main.py --base configs/ffhq/ffhq_4_scales_joint-training.yaml -t --gpus 0, 
```

## Shout-Outs
Many thanks to all who make their work and implementations publicly available. 
For this work, these were in particular:

- The extremely clear and extensible encoder-decoder transformer implementations by [lucidrains](https://github.com/lucidrains): 
https://github.com/lucidrains/x-transformers
- Emiel Hoogeboom et al's paper on multinomial diffusion and argmax flows: https://arxiv.org/abs/2102.05379

![teaser](https://raw.githubusercontent.com/CompVis/imagebart/master/assets/foxchain.png)

## References

[1] Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N. &amp; Ganguli, S.. (2015). Deep Unsupervised Learning using Nonequilibrium Thermodynamics. <i>Proceedings of the 32nd International Conference on Machine Learning

## Bibtex

```
@article{DBLP:journals/corr/abs-2108-08827,
  author    = {Patrick Esser and
               Robin Rombach and
               Andreas Blattmann and
               Bj{\"{o}}rn Ommer},
  title     = {ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive
               Image Synthesis},
  journal   = {CoRR},
  volume    = {abs/2108.08827},
  year      = {2021}
}
```