Long-CLIP

KO | EN

gitlite — search

#javascript #python #hacktoberfest #react #typescript #llm #golang #android #java #pytorch #security #ios #mcp #open-source

(★ 902)

[ECCV 2024] official code for "Long-CLIP: Unlocking the Long-Text Capability of CLIP"

File Explorer

Download Latest Version (.zip)

- model_zoo.md
- - coco.py
  - flickr30k.py
  - Urban1k.py
- demo.png
- demo_SDXL.png
- framework.PNG
- generation.png
- retrieval.png
- __init__.py
- bpe_simple_vocab_16e6.txt.gz
- longclip.py
- model_longclip.py
- simple_tokenizer.py
- - coca_base.json
  - coca_roberta-ViT-B-32.json
  - coca_ViT-B-32.json
  - coca_ViT-L-14.json
  - convnext_base.json
  - convnext_base_w.json
  - convnext_base_w_320.json
  - convnext_large.json
  - convnext_large_d.json
  - convnext_large_d_320.json
  - convnext_small.json
  - convnext_tiny.json
  - convnext_xlarge.json
  - convnext_xxlarge.json
  - convnext_xxlarge_320.json
  - EVA01-g-14-plus.json
  - EVA01-g-14.json
  - EVA02-B-16.json
  - EVA02-E-14-plus.json
  - EVA02-E-14.json
  - EVA02-L-14-336.json
  - EVA02-L-14.json
  - mt5-base-ViT-B-32.json
  - mt5-xl-ViT-H-14.json
  - nllb-clip-base-siglip.json
  - nllb-clip-base.json
  - nllb-clip-large-siglip.json
  - nllb-clip-large.json
  - RN101-quickgelu.json
  - RN101.json
  - RN50-quickgelu.json
  - RN50.json
  - RN50x16.json
  - RN50x4.json
  - RN50x64.json
  - roberta-ViT-B-32.json
  - swin_base_patch4_window7_224.json
  - ViT-B-16-plus-240.json
  - ViT-B-16-plus.json
  - ViT-B-16-quickgelu.json
  - ViT-B-16-SigLIP-256.json
  - ViT-B-16-SigLIP-384.json
  - ViT-B-16-SigLIP-512.json
  - ViT-B-16-SigLIP-i18n-256.json
  - ViT-B-16-SigLIP.json
  - ViT-B-16.json
  - ViT-B-32-256.json
  - ViT-B-32-plus-256.json
  - ViT-B-32-quickgelu.json
  - ViT-B-32.json
  - ViT-bigG-14-CLIPA-336.json
  - ViT-bigG-14-CLIPA.json
  - ViT-bigG-14.json
  - ViT-e-14.json
  - ViT-g-14.json
  - ViT-H-14-378-quickgelu.json
  - ViT-H-14-CLIPA-336.json
  - ViT-H-14-CLIPA.json
  - ViT-H-14-quickgelu.json
  - ViT-H-14.json
  - ViT-H-16.json
  - ViT-L-14-280.json
  - ViT-L-14-336.json
  - ViT-L-14-CLIPA-336.json
  - ViT-L-14-CLIPA.json
  - ViT-L-14-quickgelu.json
  - ViT-L-14.json
  - ViT-L-16-320.json
  - ViT-L-16-SigLIP-256.json
  - ViT-L-16-SigLIP-384.json
  - ViT-L-16.json
  - ViT-M-16-alt.json
  - ViT-M-16.json
  - ViT-M-32-alt.json
  - ViT-M-32.json
  - ViT-S-16-alt.json
  - ViT-S-16.json
  - ViT-S-32-alt.json
  - ViT-S-32.json
  - ViT-SO400M-14-SigLIP-384.json
  - ViT-SO400M-14-SigLIP.json
  - vit_medium_patch16_gap_256.json
  - vit_relpos_medium_patch16_cls_224.json
  - xlm-roberta-base-ViT-B-32.json
  - xlm-roberta-large-ViT-H-14.json
- __init__.py
- big_vision.py
- bpe_simple_vocab_16e6.txt.gz
- coca_model.py
- constants.py
- factory.py
- hf_configs.py
- hf_model.py
- loss.py
- model.py
- modified_resnet.py
- openai.py
- pos_embed.py
- pretrained.py
- push_to_hf_hub.py
- timm_model.py
- tokenizer.py
- transform.py
- transformer.py
- utils.py
- version.py
- zero_shot_classifier.py
- zero_shot_metadata.py
- demo_SDXL.png
- encode_prompt.py
- SDXL.md
- sdxl.py
- SDXL_img2img.py
- SDXL_pipeline.py
- scheduler.py
- sharegpt4v.py
- train.md
- train.py
- train_slurm.sh
- utils.py
demo.py
LICENSE
README.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Long-CLIP
This repository is the official implementation of Long-CLIP

**Long-CLIP: Unlocking the Long-Text Capability of CLIP**\
[Beichen Zhang](https://beichenzbc.github.io), [Pan Zhang](https://panzhang0212.github.io/), [Xiaoyi Dong](https://lightdxy.github.io/), [Yuhang Zang](https://yuhangzang.github.io/), [Jiaqi Wang](https://myownskyw7.github.io/)

## 💡 Highlights
- 🔥 **Long Input length** Increase the maximum input length of CLIP from **77** to **248**.
- 🔥 **Strong Performace** Improve the R@5 of long-caption text-image retrieval by **20%** and traditional text-image retrieval by **6%**.
- 🔥 **Plug-in and play** Can be directly applied in **any work** that requires long-text capability.

## 📜 News
🚀 [2024/7/3] Our paper has been accepted by ***ECCV2024***.

🚀 [2024/7/3] We release the code of using Long-CLIP in ***SDXL***. For detailed information, you may refer to `SDXL/SDXL.md`.

🚀 [2024/5/21] We update the paper and checkpoints after fixing the bug in DDP and add results in Urban-1k. Special thanks to @MajorDavidZhang for finding and refining this bug in DDP! Now the fine-tuning only takes ***0.5*** hours on *8 GPUs*!

🚀 [2024/5/21] Urban-1k: a scaling-up version of Urban-200 dataset in the paper has been released at this [page](https://huggingface.co/datasets/BeichenZhang/Urban1k).

🚀 [2024/4/1] The training code is released!

🚀 [2024/3/25] The Inference code and models ([LongCLIP-B](https://huggingface.co/BeichenZhang/LongCLIP-B) and [LongCLIP-L](https://huggingface.co/BeichenZhang/LongCLIP-L)) are released!

🚀 [2024/3/25] The [paper](https://arxiv.org/abs/2403.15378) is released!

## 👨‍💻 Todo
- [x] Training code for Long-CLIP based on OpenAI-CLIP
- [x] Evaluation code for Long-CLIP
- [x] evaluation code for zero-shot classification and text-image retrieval tasks.
- [x] Usage example of Long-CLIP
- [x] Checkpoints of Long-CLIP

## 🛠️ Usage

### Installation

Our model is based on [CLIP](https://github.com/openai/CLIP), please prepare environment for CLIP.

### how to use

Please first clone our [repo](https://github.com/beichenzbc/Long-CLIP) from github by running the following command.

```shell
git clone https://github.com/beichenzbc/Long-CLIP.git
cd Long-CLIP
```

Then, download the checkpoints of our model [LongCLIP-B](https://huggingface.co/BeichenZhang/LongCLIP-B) and/or [LongCLIP-L](https://huggingface.co/BeichenZhang/LongCLIP-L) and place it under `./checkpoints`

```python
from model import longclip
import torch
from PIL import Image

device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = longclip.load("./checkpoints/longclip-B.pt", device=device)

text = longclip.tokenize(["A man is crossing the street with a red car parked nearby.", "A man is driving a car in an urban scene."]).to(device)
image = preprocess(Image.open("./img/demo.png")).unsqueeze(0).to(device)

with torch.no_grad():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    
    logits_per_image = image_features @ text_features.T
    probs = logits_per_image.softmax(dim=-1).cpu().numpy()

print("Label probs:", probs) 
```

### Evaluation
#### Zero-shot classification

To run zero-shot classification on imagenet dataset, run the following command after preparing the data
```shell
cd eval/classification/imagenet
python imagenet.py
```

Similarly, run the following command for cifar datset
```shell
cd eval/classification/cifar
python cifar10.py               #cifar10
python cifar100.py              #cifar100
```

#### Retrieval
To run text-image retrieval on COCO2017 or Flickr30k, run the following command after preparing the data
```shell
cd eval/retrieval
python coco.py                  #COCO2017
python flickr30k.py             #Flickr30k
```
### Traning
Please refer to `train/train.md` for training details.

## ⭐ Demos
### Long-CLIP-SDXL
 <a> 
<img src="./img/demo_SDXL.png" width="900" />
</a>

### Long-caption text-image retrieval 
 <a> 
<img src="./img/retrieval.png" width="900" />
</a>

### Plug-and-Play text to image generation 
 <a> 
<img src="./img/generation.png" width="900" />
</a>

## Citation
If you find our work helpful for your research, please consider giving a citation:
```
@article{zhang2024longclip,
        title={Long-CLIP: Unlocking the Long-Text Capability of CLIP},
        author={Beichen Zhang and Pan Zhang and Xiaoyi Dong and Yuhang Zang and Jiaqi Wang},
        journal={arXiv preprint arXiv:2403.15378},
        year={2024}
}
```

Long-CLIP

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Long-CLIP

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Download README