AnyText2 - GitLite

KO | EN

gitlite — search

#javascript #hacktoberfest #react #ai #typescript #golang #machine-learning #rust #docker #nodejs #windows #mcp #openai #open-source

AnyText2

(★ 215)

Official implementation code of the paper <AnyText2: Visual Text Generation and Editing With Customizable Attributes>

파일 탐색기

최종 버전 다운로드 (.zip)

- cldm.py
- ddim_hacked.py
- embedding_manager.py
- hack.py
- logger.py
- model.py
- recognizer.py
- eval.jpg
- framework.jpg
- sample.jpg
- anytext2_multiGPUs.py
- anytext2_singleGPU.py
- clipscore.py
- eval_clip.sh
- eval_dgocr.py
- eval_fid.sh
- eval_ocr.sh
- gen_imgs_anytext2.sh
- bangqiumao.png
- banner2.jpg
- cake.png
- edit10.png
- edit11.png
- edit12.png
- edit13.png
- edit14.png
- edit15.png
- edit16.png
- edit2.png
- edit3.png
- edit4.png
- edit5.png
- edit6.png
- edit7.png
- edit8.png
- edit9.png
- gen1.png
- gen10.png
- gen11.png
- gen12.png
- gen13.png
- gen14.png
- gen15.png
- gen16.png
- gen17.png
- gen18.png
- gen19.png
- gen2.png
- gen20.png
- gen21.png
- gen3.png
- gen4.png
- gen5.png
- gen6.png
- gen7.png
- gen8.png
- gen9.png
- ref10.jpg
- ref11.jpg
- ref12.png
- ref13.jpg
- ref14.png
- ref15.jpeg
- ref16.jpeg
- ref2.jpg
- ref3.jpg
- ref4.jpg
- ref5.jpg
- ref6.jpg
- ref7.jpg
- ref8.jpg
- ref9.jpg
- tiger.png
- yuebing.png
- - .DS_Store
  - Arial_Unicode.ttf
- - Arial_Unicode.ttf
  - BadScript-Regular.ttf
  - BodoniModa-Italic-VariableFont_opsz,wght.ttf
  - IndieFlower-Regular.ttf
  - Jaini-Regular.ttf
  - LongCang-Regular.ttf
  - Pacifico-Regular.ttf
  - PlayfairDisplay-VariableFont_wght.ttf
  - SourceHanSansCN-Medium.otf
  - 仿乾隆字体.ttf
  - 淘宝买菜营销体.otf
  - 站酷小薇LOGO体.otf
  - 站酷庆科黄油体.ttf
  - 站酷快乐体2016修订版.ttf
  - 钉钉进步体.ttf
  - 阿里妈妈东方大楷.otf
  - 黄引齐招牌体.ttf
- .DS_Store
- Arial_Unicode.ttf
- bg_noise.png
- lang_font_dict.npy
- - __init__.py
  - util.py
- - autoencoder.py
- - attention.py
  - ema.py
- util.py
- anytext2_sd15.yaml
- common.py
- en_dict.txt
- ppocr_keys_v1.txt
- RecCTCHead.py
- RecModel.py
- RecMv1_enhance.py
- RecSVTR.py
- RNN.py
- en_dict.txt
- ppocr_keys_v1.txt
.gitignore
bert_tokenizer.py
dataset_util.py
demo.py
environment.yaml
LICENSE
lora_util.py
ms_wrapper.py
README.md
style.css
t3_dataset.py
tool_add_anytext.py
train.py
util.py

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# AnyText2: Visual Text Generation and Editing With Customizable Attributes

<a href='https://arxiv.org/abs/2411.15245'><img src='https://img.shields.io/badge/Paper-Arxiv-red'></a> <a href='https://github.com/tyxsspa/AnyText2'><img src='https://img.shields.io/badge/Code-Github-green'></a> <a href='https://modelscope.cn/studios/iic/studio_anytext2'><img src='https://img.shields.io/badge/Demo-ModelScope-lightblue'></a> <a href='https://huggingface.co/spaces/modelscope/AnyText2'><img src='https://img.shields.io/badge/Demo-HuggingFace(TODO)-yellow'></a>

![sample](https://raw.githubusercontent.com/tyxsspa/AnyText2/main/docs/sample.jpg "sample")

## 📌News
[2025.03.01] - 🧨We release all code, checkpoint and dataset of AnyText2, and the online demo is available on [ModelScope](https://modelscope.cn/studios/iic/studio_anytext2)!  
[2024.11.22] - The paper is available at [arxiv](https://arxiv.org/abs/2411.15245).

## ⏰TODOs

- [ ] Release SDXL version of AnyText2 (AnyText2XL)
- [x] Release inference/training/evaluation code, checkpoint, dataset of AnyText2

## 💡Methodology
AnyText2 is a novel method that enables precise control over multilingual text attributes in natural scene image generation and editing. Our approach consists of two main components. First, we propose a WriteNet+AttnX architecture that injects text rendering capabilities into a pre-trained T2I model.  Compared to its predecessor, AnyText, our new approach not only enhances image realism but also achieves a 19.8% increase in inference speed.  Second, we explore techniques for extracting fonts and colors from scene images and develop a Text Embedding Module that encodes these text attributes separately as conditions. As an extension of AnyText, this method allows for customization of attributes for each line of text, leading to improvements of 3.3% and 9.3% in text accuracy for Chinese and English, respectively. Through comprehensive experiments, we demonstrate the state-of-the-art performance of our method.

![framework](https://raw.githubusercontent.com/tyxsspa/AnyText2/main/docs/framework.jpg "framework")

## 🛠Installation
```bash
# Download checkpoint (pip install modelscope first)
from modelscope import snapshot_download
your_path_to_model_dir = snapshot_download('iic/cv_anytext2')
# Clone anytext code and move checkpoint
git clone https://github.com/tyxsspa/AnyText2.git
cd AnyText2 && mkdir -p models  &&  sleep 1 && mv your_path_to_model_dir/* models
# Create a new environment and install packages as follows:
conda env create -f environment.yaml
conda activate anytext2
```

## 🔮Inference

**[Recommend]**： We release a demo on [ModelScope](https://modelscope.cn/studios/iic/studio_anytext2)

If you have advanced GPU (with at least 8G memory), it is recommended to deploy our demo as below, which includes usage instruction, user interface and abundant examples.
```bash
python demo.py
```
The other parameters of run the demo are consistent with [AnyText](https://github.com/tyxsspa/AnyText?tab=readme-ov-file#inference).

## 📈Evaluation

The evaluation scripts are located in `AnyText2/eval`, and their usage remains consistent with [AnyText](https://github.com/tyxsspa/AnyText?tab=readme-ov-file#evaluation). We have also updated the evaluation files for the long caption version in [AnyText-benchmark](https://modelscope.cn/datasets/iic/AnyText-benchmark/summary).

We trained the **AnyText2XL** model based on diffusers, using the SDXL model and 1024x1024 version of AnyWord-3M dataset, it will be released soon.
![eval](https://raw.githubusercontent.com/tyxsspa/AnyText2/main/docs/eval.jpg "eval")

## 🚂Training

We have updated the training dataset [AnyWord-3M](https://modelscope.cn/datasets/iic/AnyWord-3M), where the image data remains the same, only annotation files for training AnyText2 are added. These files include long captions and text color labels. You can start the training by running `python train.py`, and the details are consistent with the documentation for [AnyText](https://github.com/tyxsspa/AnyText?tab=readme-ov-file#training).

## Citation
```
@article{tuo2023anytext,
      title={AnyText: Multilingual Visual Text Generation And Editing}, 
      author={Yuxiang Tuo and Wangmeng Xiang and Jun-Yan He and Yifeng Geng and Xuansong Xie},
      year={2023},
      eprint={2311.03054},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
```

```
@misc{tuo2024anytext2,
      title={AnyText2: Visual Text Generation and Editing With Customizable Attributes}, 
      author={Yuxiang Tuo and Yifeng Geng and Liefeng Bo},
      year={2024},
      eprint={2411.15245},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
}
```