GOAT-PEFT - GitLite

KO | EN

gitlite — search

#python #hacktoberfest #react #typescript #linux #docker #nodejs #macos #windows #kubernetes #vue #pytorch #openai #open-source

GOAT-PEFT

(★ 148)

[ICML2025] Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment

파일 탐색기

최종 버전 다운로드 (.zip)

- - AddSub.json
  - addsub_1.json
  - test.json
- - AQuA.json
  - aqua_1.json
  - test.json
- - test.json
  - train.json
- - test.json
  - train.json
- - test.json
  - train.json
- - gsm8k.json
  - gsm8k_1.json
  - test.json
- - test.json
  - train.json
- - test.json
- - data_process.py
  - test.json
  - testset.json
  - trainset.json
  - validset.json
- - MultiArith.json
  - multiarith_1.json
  - test.json
- - test.json
  - train.json
- - test.json
  - train.json
- - SingleEq.json
  - singleeq_1.json
  - test.json
- - test.json
  - train.json
- - SVAMP.json
  - svamp_1.json
  - test.json
- - test.json
  - train.json
- - download_dataset.sh
  - process_dtd.py
  - process_euro.py
  - process_sun.py
- .DS_Store
- commonsense_170k.json
- - deepspeed_zero2.json
- - __init__.py
- download_data.py
- eval_nlg.py
- train_nlg.py
- train_nlu.py
- train_vit.py
- main.png
.gitignore
chat.sh
code.sh
cr.sh
glue.sh
LICENSE
math.sh
README.md
requirements.txt
vit.sh

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# GOAT: a new LoRA Mixture-of-Experts Framework🚀

<div align="center">
<p align="center" style="font-size: larger;">
<strong> [ICML 2025] </strong> <a href="https://arxiv.org/pdf/2502.16894v3"> Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment</a>
</p>

![main](https://raw.githubusercontent.com/Facico/GOAT-PEFT/master/img/main.png)
</div>

Official implementation of **GOAT**: a novel framework that boosts Low-Rank Adaptation (LoRA) for large-scale model fine-tuning by integrating adaptive singular-value priors and theoretical optimization alignment within a Mixture-of-Experts (MoE) architecture.

Our Paper has been accepted by ICML2025.

## 🚀Highlights

- Adaptive Priors Initialization: Segment the pre-trained weight’s SVD into multiple chunks; initialize each MoE expert with a distinct singular-value segment, dynamically selected by the router.

- Theoretical Optimization Alignment: Derive closed-form scaling factors and residual corrections to align LoRA’s low-rank adaptation gradients and weights with full fine-tuning trajectories.

- State-of-the-Art Performance: Outperforms existing LoRA and LoRA-MoE baselines across 25 benchmarks in image classification, NLU, NLG, and commonsense reasoning, closing or exceeding full fine-tuning accuracy.

## Installation

Follow the steps below to set up the environment and install the necessary dependencies:

1. **Create a new Conda environment**:
   ```bash
   conda create --name goat python=3.9
   conda activate goat
   ```

2. **Install required dependencies**:
   Make sure you have `pip` installed, then run:
   ```bash
   pip install -r requirements.txt
   pip install flash-attn==2.7.3
   pip install human_eval
   ```

## Training

### Download Datasets

Most of the data is built into the `dataset/` directory. CV-related data needs to be downloaded separately, and the download method can be found in the corresponding training script.
- Since it is necessary to download data from Kaggle, the Kaggle key needs to be properly set in dataset/~data/download_dataset.sh.

### Training Scripts

- **Image Classification**:  
  Train on image classification datasets using Vision Transformers (ViTs):  
  ```bash
  bash vit.sh
  ```

- **Natural Language Generation (NLG)**:  
  Fine-tune for generation tasks, such as mathematical reasoning, code generation, or chatbot tasks:  
  ```bash
  bash math.sh   # For math reasoning tasks
  bash code.sh   # For code generation tasks
  bash chat.sh   # For chatbot tasks
  ```

- **Natural Language Understanding (NLU)**:  
  Fine-tune on NLU tasks, such as the GLUE benchmark:  
  ```bash
  bash glue.sh
  ```

- **Commonsense QA (CR)**:  
  Fine-tune on CR tasks:  
  ```bash
  bash cr.sh
  ```
---

## Evaluation

In Training, our training script will automatically evaluate the downstream tasks (Performance log on Wandb), except for chat task.

For chat task, we use [FastChat](https://github.com/lm-sys/FastChat) to generation and evaluate with GPT-4, please read their instruction.

---

## Citation

If you find our work helpful, please consider citing our paper:
```bash
@misc{fan2025makeloragreatagain,
      title={Make LoRA Great Again: Boosting LoRA with Adaptive Singular Values and Mixture-of-Experts Optimization Alignment}, 
      author={Chenghao Fan and Zhenyi Lu and Sichen Liu and Xiaoye Qu and Wei Wei and Chengfeng Gu and Yu Cheng},
      year={2025},
      eprint={2502.16894},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.16894}, 
}
```

## Contact
If you have any question, feel free to contact 📫 facicofan@gmail.com or luzhenyi529@gmail.com

## References
The code refers to the repo [LoRAPro](https://github.com/mrflogs/LoRA-Pro)