Finetune-Qwen2.5-VL - GitLite

KO | EN

gitlite — search

#hacktoberfest #golang #machine-learning #deep-learning #linux #docker #nodejs #java #awesome #cli #kubernetes #vue #mcp #openai

Finetune-Qwen2.5-VL

(★ 170)

Fine-tuning Qwen2.5-VL for vision-language tasks | Optimized for Vision understanding | LoRA & PEFT support.

File Explorer

Download Latest Version (.zip)

- accelerate.yaml
- vlm_config.yaml
- vlm_inference_config.yaml
- vlm_merge_adapter_config.yaml
- dataset_info.json
- evaluation.py
- - __init__.py
- __init__.py
- inference.py
- merge_model.py
- train.py
- version.py
- __init__.py
- test_data_handler.py
- __init__.py
- constants.py
- logger.py
- misc.py
- packages.py
- ploting.py
.dockerignore
.gitignore
.pre-commit-config.yaml
__init__.py
Dockerfile
LICENSE
main.py
README.md
requirements.txt
setup.py

# Installation Guide

1. Get the code

git clone https://github.com/sandy1990418/Finetune-Qwen2.5-VL

Downloads the entire project code from GitHub to your computer.

cd Finetune-Qwen2.5-VL

Moves into the project folder you just downloaded.

2. Docker

Easy Recommended

Prerequisites

Git Needed to download the project code from GitHub.
Docker Desktop Needed to build and run containers. Install it and keep it running in the background.

docker build --no-cache -t vlm_finetune:latest .

Builds a runnable image based on the Dockerfile.

docker run -it --name CONATINER_NAME -v LOCAL_PATH:/VLM vlm_finetune:latest

Runs the built image as an actual container.

✅ Run docker compose ps to check the containers are Up. If the README mentions a port, open http://localhost:PORT in your browser.

Pulled directly from this repo's README.

3. Python

Easy

Prerequisites

Git Needed to download the project code from GitHub.
Python 3 On Windows, be sure to check 'Add Python to PATH' during installation.

pip install -r requirements.txt

Installs the Python libraries listed in requirements.txt (or similar).

✅ If it runs without errors and prints output in the terminal, it worked.

Pulled directly from this repo's README.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Finetune-Qwen2.5-VL

A comprehensive toolkit for finetuning the Qwen2.5-VL (Visual Language) model using LoRA. This project provides easy-to-use scripts for Supervised Fine-Tuning (SFT), LoRA merging, and inference.

## Requirements

```bash
# Core dependencies
torch>=2.0.0
transformers>=4.37.0
peft>=0.7.0
accelerate>=0.21.0
```

## Installation

```bash
git clone https://github.com/sandy1990418/Finetune-Qwen2.5-VL.git
cd Finetune-Qwen2.5-VL
pip install -r requirements.txt
```

## Usage

### 1.1 Supervised Fine-Tuning (SFT) - Single GPU

Run the following command to start the fine-tuning process:

```bash
python src/train.py config/vlm_config.yaml

or

python main.py config/vlm_config.yaml
```

The `vlm_config.yaml` should contain your training configurations such as:
- Model parameters
- Training hyperparameters
- Dataset configurations
- LoRA settings

Check the configurations `distributed_type: "NO"` in `accelerate.yaml`.

### 1.2 Supervised Fine-Tuning (SFT) - Multiple GPU

Run the following command to start the fine-tuning process:

```bash
python main.py config/vlm_config.yaml config/accelerate.yaml
```

The `vlm_config.yaml` should contain your training configurations and  `accelerate.yaml` should contain `accelerate` configurations. If you want to use multiple GPUs, set `use_accelerate: true` in `vlm_config.yaml` and `distributed_type: "MULTI_GPU", gpu_ids: all` in `accelerate.yaml`

A better approach would be to use Ray, but I am currently facing some issues that have yet to be resolved. I plan to work on this further in the future.

### 2. Merge LoRA Weights

After training, merge the LoRA weights with the base model:

```bash
python src/merge_model.py config/vlm_merge_adapter_config.yaml
```

### 3. Inference

Run inference with your fine-tuned model:

```bash
python src/inference.py config/vlm_inference_config.yaml
```

### 4. Evaluation
```bash
python evaluation/evaluation.py config/vlm_inference_config.yaml
```

### 5. Dockerfile
```bash
docker build --no-cache -t vlm_finetune:latest .
docker run -it --name CONATINER_NAME -v LOCAL_PATH:/VLM vlm_finetune:latest
```

## TODO
- [ ] Resolve issues with Ray for multi-GPU training
- [ ] Implement evaluation pipeline for fine-tuned models  
- [ ] Add test cases for training, merging, and inference  
- [ ] Load Data may be more flexible
- [ ] Load Data support Image_url in Trainig stage
- [ ] Finetune JSON Dataset

## Contributing

1. Fork the repository
2. Create your feature branch (`git checkout -b feature/amazing-feature`)
3. Commit your changes (`git commit -m 'Add some amazing feature'`)
4. Push to the branch (`git push origin feature/amazing-feature`)
5. Open a Pull Request

## Acknowledge
This project is based on [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory/tree/main). Special thanks to the original authors and contributors!