Image-to-Prompts

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

Image-to-Prompts

★ 11 Open GitHub ↗

A generative text-to-image model is a model that can generate an image from a text prompt.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Image to Prompt

| [Code](./clipinterrogator-ofa-vit.ipynb) | [Slide](https://www.slideshare.net/jacksonChen22/imagetopromptspdf) | [Report](./Image_To_Prompts.pdf) |

A generative text-to-image model is a model that can generate an image from a text prompt.

This repository is a final project for the course [EECM30064 Deep Learning](https://timetable.nycu.edu.tw/?r=main/crsoutline&Acy=111&Sem=2&CrsNo=535361&lang=zh-tw)

## Contributors

## Motivation and Background

[Stable Diffusion - Image to Prompts](https://www.kaggle.com/competitions/stable-diffusion-image-to-prompts/overview) is a competition on Kaggle.

The goal of this competition is to reverse the typical direction of a generative text-to-image model: instead of generating an image from a text prompt.

We want to  create a model which can predict the text prompt given a generated image. And making predictions on a dataset containing a wide variety of $\verb|(prompt, image)|$ pairs generated by Stable Diffusion 2.0, in order to understand how reversible the latent relationship is.

Sample images from the competition dataset and their corresponding prompts are shown below.

<table>
    <tr>
        <th>
            <center>Image</center>
        </th>
        <th>
            <center>Prompt</center>
        </th>
    </tr>
    <tr>
        <td>
            <center><img src="./images/92e911621.png" width="200" height="200"></center>
        </td>
        <td>
            <center><code>ultrasaurus holding a black bean taco in the woods, near an identical cheneosaurus</code></center>
        </td>
    </tr>
</table>

## Methodology

Our method is to ensemble the CLIP Interrogator, OFA model, and ViT model.

Here's the ratio for three different model
- Vision Transformer (ViT) model:  74.88%
- CLIP Interrogator: 21.12%
- OFA model fine-tuned for image captioning: 4%

## Application and Datasets

### Application

Based on the Kaggle competition, we want to build a model to predict the prompts that were used to generate target images.

### Datasets

Prompts for this challenge were generated using a variety of (non disclosed) methods, and range from fairly simple to fairly complex with multiple objects and modifiers.

Images were generated from the prompts using Stable Diffusion $2.0$ ($768$-v-ema.ckpt) and were generated with 50 steps at $768 \times 768$ px and then downsized to $512 \times 512$ for the competition dataset. The hidden re-run test folder contains approximately $16,000$ images.

## References

[1] [Learning Transferable Visual Models From Natural Language Supervision](https://arxiv.org/pdf/2103.00020.pdf)

[2] [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://arxiv.org/pdf/2010.11929.pdf)

[3] [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://arxiv.org/pdf/1409.1556.pdf)

[4] [SentenceTransformers](https://www.sbert.net/)

[5] [CLIPInterrogator + OFA + ViT](https://www.kaggle.com/code/motono0223/clipinterrogator-ofa-vit)

[6] [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://arxiv.org/pdf/2103.14030.pdf)

[7] [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://arxiv.org/pdf/2205.01917.pdf)

Image-to-Prompts

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README