EyecareGPT

KO | EN

gitlite — search

#javascript #java #python #chrome-extension #bioinformatics #nodejs #node #plugin #php #api #rest #go

★ 73 Open GitHub ↗

【ACM MM 2025】Official Repo for Paper ‘’EyecareGPT: Boosting Comprehensive Ophthalmology Understanding with Tailored Dataset, Benchmark and Model‘’

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align = "center">
<img src="images/logo.png" alt="icon" style="width:50px; vertical-align:middle;" />
EyecareGPT: Boosting Comprehensive Ophthalmology
Understanding with Tailored Dataset, Benchmark and Model
</h1>

<div align="center">
Sijing Li1, Tianwei Lin1, Lingshuai Lin2, Wenqiao Zhang1, Jiang Liu1, Xiaoda Yang1, Juncheng Li1, Yucheng He3, Song xiaohui1, Jun Xiao1, Yueting Zhuang1, Beng Chin Ooi4

1Zhejiang University,
2Harbin Institute of Technology,
3Chenzhou No. 1 People's Hospital,
4National University of Singapore

</div>

## 🌟 Overview

Welcome to **EyecareGPT!** 🚀
This repository showcases the **Eyecare Kit**, a comprehensive solution designed to advance the adaptability and performance of Large Vision-Language Models (LVLMs) in the specialized domain of ophthalmology. Our work aims to  facilitate research on intelligent ophthalmic diagnosis field by introducing innovations across three key aspects: **Dataset**, **Benchmark**, and **Model**.

##### :clap: Key Highlights:

- **Eyecare-100K: The First Comprehensive Ophthalmic Visual Instruction Dataset.** We present a large-scale, diverse dataset comprising approximately **102,000** visual question answering (VQA) pairs, covering **8** imaging modalities, **3** tasks， over **15** anatomical structures, and more than **100** types of eye diseases.
- **Eyecare-Bench: A Systematic Benchmark for Intelligent Ophthalmic Diagnosis.** We introduce a novel benchmark for in-depth evaluation of Med-LVLMs on closed QA, open QA, and report generation tasks, featuring around 15,000 carefully selected examples.
- **EyecareGPT: An Adapted LVLM Architecture for Ophthalmology.** We present a model incorporating high-resolution visual feature extraction, an adaptive resolution mechanism, and dense feature integration, with scalable variants for real-world deployment.

## 🔥 News

- **[2025.02.17]** We have released the pre-trained weight on HuggingFace and dataset demo of Eyecare-100K.
- **[now]** We are building our EyecareGPT-Plus specialized for temporal ocular data such as FFA/ICGA!.

### TODO

- [x] Release  pre-trained weight of the model.
- [x] Release the demo of  Eyecare-100K.
- [x] Release inference code (e.g. training scripts).

## 📚 Task Classification and Support

**Eyecare-100K** aggregates **real-world** ophthalmic data across **8** modalities, **15+** anatomical structures and **100+** eye diseases, supporting multi-modal report generation and fine-grained visual QA tasks.

## :mailbox: Data collection and statistics

We develop an automated **multi-agent data engine** to create Eyecare-100K, converting categorized labels and raw reports into 3 kinds of structured VQA pairs: closed-QA, open-QA, report generation.

## :ferris_wheel: Architecture

**EyecareGPT**: We introduce an LVLM architecture adapted to complex, heterogeneous ophthalmic clinical imaging, achieving SoTA performance. **(i) High-Resolution Feature Extraction:** Employ SigLIP to enhance local lesion perception.  **(ii) Adaptive Resolution Mechanism:** Accommodate variable resolutions in clinical ophthalmic imaging. **(iii) Layer-wise Dense Connector (LDC):** Densely integrate multi-scale visual features and preserve fine-grained structural information.

## 🛠️ Getting Started

**EyecareGPT**  is our ophthalmology-specialized LVLM, built on **Eyecare-100K**. We have released our model in two configurations, **EyecareGPT-3.8B** and **EyecareGPT-7B**, to suit different requirements and resource availability:

- EyecareGPT-3.8B: A smaller version optimized for speed and reduced memory usage.
- EyecareGPT-7B: A larger version designed for higher Performance.

**Model Access：**

EyecareGPT utilizes `siglip-so400m-patch14-384` as the visual encoder and employs `Phi-3.5-mini-instruct` and `Qwen2.5-VL-7B-Instruct` as the pre-trained LLM base models for EyecareGPT-3.8B and EyecareGPT-7B, respectively. Please download the corresponding weights:

## 🔗 Citation
If you found this work useful, please consider giving this repository a star and citing our paper as followed:
```
@misc{li2025eyecaregptboostingcomprehensiveophthalmology,
      title={EyecareGPT: Boosting Comprehensive Ophthalmology Understanding with Tailored Dataset, Benchmark and Model}, 
      author={Sijing Li and Tianwei Lin and Lingshuai Lin and Wenqiao Zhang and Jiang Liu and Xiaoda Yang and Juncheng Li and Yucheng He and Xiaohui Song and Jun Xiao and Yueting Zhuang and Beng Chin Ooi},
      year={2025},
      eprint={2504.13650},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2504.13650}, 
}
```

EyecareGPT

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README