GPT-ImgEval - GitLite

KO | EN

gitlite — search

#python #hacktoberfest #ai #android #rust #linux #nodejs #java #windows #vue #pytorch #ios #openai #open-source

GPT-ImgEval

(★ 307)

GPT-ImgEval: Evaluating GPT-4o’s state-of-the-art image generation capabilities

파일 탐색기

최종 버전 다운로드 (.zip)

- EvalScore_bar.jpg
- GenEval.jpg
- Geneval.jpg
- GenEval_cases.jpg
- GPT_Structure_Detection.jpg
- logo.jpg
- multi-turn.jpg
- multi-turn2.jpg
- pipeline.jpg
- smartedit_case-2.jpg
- SmartEdit_Fig1.jpg
- Structure.jpg
- WISE-New.jpg
- WISE.jpg
- WISE_case.jpg
- - config.json
  - prompt.txt
- - config.json
  - prompts.txt
- - app_controller.cpython-312.pyc
  - config_manager.cpython-312.pyc
  - file_manager.cpython-312.pyc
  - image_processor.cpython-312.pyc
  - processor.cpython-312.pyc
- app_controller.py
- config_manager.py
- file_manager.py
- image_processor.py
- processor.py
chatgpt_script.py
get_position.py
README.md
requirements.txt

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/PicoTrex/GPT-ImgEval

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd GPT-ImgEval

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

pip install -r requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

파이썬 스크립트(또는 모듈)를 실행합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generation

<a href="https://arxiv.org/abs/2504.02782" target="_blank"><img src="https://img.shields.io/badge/arXiv-arXiv-red?style=badge&logo=arXiv" alt="Paper PDF" height="25"></a>
<a href='https://huggingface.co/datasets/Yejy53/GPT-ImgEval'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Dataset-yellow' height="25"></a>

![method](https://raw.githubusercontent.com/PicoTrex/GPT-ImgEval/main/assets/pipeline.jpg)

## 📰 News

* **[2025.5.2]**  💡 We have updated the arxiv with more complete GenEval test results and complete WISE dataset test results.

* **[2025.4.6]**  🚀 We have improved and uploaded the **automation script**, which supports automatic generation of images based on text prompt or image + text prompt using GPT-4o, and provides automatic downloading.

* **[2025.4.3]**  🔥 We have released **GPT-ImgEval: A First and Comprehensive Benchmark for Diagnosing GPT4o in Image Generation**. Check out the **[** [Paper](https://arxiv.org/pdf/2504.02782); [Dataset](https://huggingface.co/datasets/Yejy53/GPT-ImgEval); [Code](https://github.com/PicoTrex/GPT-ImgEval) **]**. Since GPT-4o currently lacks an image generation API, **we provide an automated script** that interacts with its web interface to support batch data processing.

## 🏆 Contributions

* 📚 **GPT-4o Image Generation Evaluation:** GPT-4o excels in image generation, surpassing previous models across benchmarks like text-to-image (GenEval), editing (Reason-Edit), and world knowledge-informed generation (WISE).
* 🔧 **Detailed Analysis:** We explore GPT-4o's generation strengths and weaknesses, compare its multi-round editing capabilities with Gemini 2.0 Flash, and assess image detectability for safety.
* 💡 **Architecture Analysis:** Our analysis suggests GPT-4o uses a Diffusion architecture for its decoder, with a potential speculative model for its encoder.

## 🔥 Evaluation Results

According to the table, GPT4o achieves the highest overall score of 0.84, largely outperforming both the frozen text encoder methods and the LLM/MLLM-enhanced approaches.

![Table1](https://raw.githubusercontent.com/PicoTrex/GPT-ImgEval/main/assets/GenEval.jpg)

Figure presents qualitative examples of GPT-4o's compositional text-to-image generation capabilities across six core evaluation categories in the GenEval benchmark.
![Geneval](https://raw.githubusercontent.com/PicoTrex/GPT-ImgEval/main/assets/GenEval_cases.jpg)

As shown in the bar chart , GPT-4o significantly outperforms all existing image editing methods on the Reason-Edit benchmark, achieving a remarkable score of **0.929**. This represents a substantial leap of **+0.357** over the best-performing method prior to 2025 (SmartEdit, **0.572**), highlighting the model’s powerful instruction-following ability and fine-grained editing control.

<img src="assets/EvalScore_bar.jpg" alt="Table2" style="display:block; margin: 0 auto; width:50%;">

GPT-4o significantly outperforms existing specialized T2I generation methods and unified MLLM-based approaches in terms of overall WiScore. GPT-4o combines exceptional world knowledge understanding with high-fidelity image generation, demonstrating a dual strength in multimodal generation tasks.

![Table3](https://raw.githubusercontent.com/PicoTrex/GPT-ImgEval/main/assets/WISE_case.jpg)

## 💻 Automated GPT-4o Script (for Batch Data Processing)

The script supports **text-only** and **text+image** inputs, and it also supports automatically **saving the output images** and **response results**.

### Prerequisites

* macOS with M1/M2/M3/M4 chip
* [ChatGPT desktop app](https://chatgpt.com/download) installed

### Usage

* Since the window positions may vary for different users, it is recommended to first **obtain the approximate position** where the image appears, and then modify the obtained **x and y coordinates** in the *config.json* file to ensure proper functionality.

( Input an example in a window and use ***get_position.py*** to get the approximate position where the image appears. For more accurate positioning, we recommend scrolling the interface to the very bottom. The scroll parameters can also be set in the scroll_amount field of the *config.json* file. )

```bash
python get_position.py
```

* After modifying the *config.json*, you can run our sample code to try out the features of our script.

```bash
# text-only example
python chatgpt_script.py --config_path example/text-only

# text-image example 
# Please change the image_folder parameter in the config.json file under the example/text-image folder to the absolute path on your computer.
python chatgpt_script.py --config_path example/text-image
```

### Troubleshooting

If the tool isn't functioning correctly:

* Make sure ChatGPT app is installed and you're logged in.
* Verify that all required permissions have been granted.
* Make sure your current input method is set to English.
* Make sure the path of the image folder is an absolute path (using a relative path often leads to image input errors).

If you set a reasonable time interval based on the GPT-4o Pro account, you will rarely get synthesis failure results.

## 🤗 Dataset Download

We uploaded the GPT generation results for the Reason-Edit and GenEval datasets, which can be downloaded 📁 [here](https://huggingface.co/datasets/Yejy53/GPT-ImgEval).

We will also upload our automated script within these two days to facilitate more people to evaluate GPT-4o's image synthesis capabilities.

![smartedit](https://raw.githubusercontent.com/PicoTrex/GPT-ImgEval/main/assets/smartedit_case-2.jpg)

## 🤔 Potential Architecture Analysis of GPT-4o

Beyond benchmark evaluations, we conduct deeper analyses to uncover GPT-4o's potential architectural choices. Specifically, we propose a model-based classification method, where a standard binary classifier is trained to distinguish between images generated by the two paradigms, and then applied to GPT-4o's outputs. Interestingly, the classifier consistently classifies GPT-4o's images as diffusion-based, providing empirical evidence that GPT-4o may internally use a diffusion head for image decoding. We also provide a complete structure deduction of GPT-4o; please refer to our paper for details.

![Classified](https://raw.githubusercontent.com/PicoTrex/GPT-ImgEval/main/assets/GPT_Structure_Detection.jpg)

![Structure](https://raw.githubusercontent.com/PicoTrex/GPT-ImgEval/main/assets/Structure.jpg)

## ❤️ Acknowledgements

We would like to thank the following open-source projects and research works:

* [GenEval](https://github.com/djghosh13/geneval)
* [SmartEdit](https://github.com/TencentARC/SmartEdit)
* [WISE](https://github.com/PKU-YuanGroup/WISE)
* [claude-chatgpt-mcp](https://github.com/syedazharmbnr1/claude-chatgpt-mcp)
* [LLM-DepthEval](https://github.com/JiahaoZhang-Public/LLM-DepthEval)
* [awesome-framework-gallery](https://github.com/LongHZ140516/awesome-framework-gallery)

## 📕 BibTeX

```bib
@article{yan2025gpt-imgeval,
      title={GPT-ImgEval: A Comprehensive Benchmark for Diagnosing GPT4o in Image Generation}, 
      author={Zhiyuan Yan and Junyan Ye and Weijia Li and Zilong Huang and Shenghai Yuan and Xiangyang He and Kaiqing Lin and Jun He and Conghui He and Li Yuan},
      journal={arXiv preprint arXiv:2504.02782},
      year={2025},
}
```