Edge-LLM

KO | EN

gitlite — search

#typescript #ai-agents #ai #dsh-plugin #deepseek-harness #open-source #cli #claude-code #codex #developer-tools #react #windows

★ 95 Open GitHub ↗

[DAC 2024] EDGE-LLM: Enabling Efficient Large Language Model Adaptation on Edge Devices via Layerwise Unified Compression and Adaptive Layer Tuning and Voting

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# EDGE-LLM: Enabling Efficient Large Language Model Adaptation on Edge Devices via Layerwise Unified Compression and Adaptive Layer Tuning and Voting

[![License: Apache 2.0](https://img.shields.io/badge/License-Apache%202.0-green)](https://opensource.org/licenses/Apache-2.0)

Zhongzhi Yu1, Zheng Wang1, Yuhan Li1, Haoran You1, Ruijie Gao1, Xiaoya Zhou3, Sreenidhi Reedy Bommu1, Yang (Katie) Zhao2, Yingyan (Celine) Lin1

*1 Georgia Institute of Technology, *2 University of Minnesota, Twin Cities*, *3 University of California, Santa Barbara**

Accepted by **[DAC 2024](https://61dac.conference-program.com/)**

The official implementation of "Edge-LLM:  Enabling Efficient Large Language Model Adaptation on Edge Devices via Layerwise Unified Compression and Adaptive Layer Tuning and Voting".

## Overview
We introduce a computation- and memory-efficient LLM tuning framework, called Edge-LLM, to facilitate affordable and effective LLM adaptation on edge devices. Specifically, Edge-LLM features three core components: (1) a layer-wise unified compression (LUC) technique to reduce the computation overhead by generating layer-wise pruning sparsity and quantization bit-width policies, (2) an adaptive layer tuning and voting scheme to reduce the memory overhead by reducing the backpropagation depth, and (3) a complementary hardware scheduling strategy to handle the irregular computation patterns introduced by LUC and adaptive layer tuning, thereby achieving efficient computation and data movements.

## Installation
To run the code, please install the dependencies using

``` bash
pip install -r requirements.txt 
```

## Training and Evaluation

### Layerwise Unified Compressed and Adaptive Layer Tuning
To launch the training of the whole Edge-LLM algorithm, please use the following command:

``` bash
bash ./scripts/edge_llm_train.sh
```

We also provide the script to run each enablers of our proposed framework below

### Quantize Model
In our implementation, we build our quantization mmethod on top of [LLM-QAT](https://github.com/facebookresearch/LLM-QAT). To try our proposed layer-wise pruning technique to prune the model, please use the following command to quantize and tune the model:

``` bash
bash ./scripts/layer_wise_quantization.sh
```

### Prune Model
In our implementation, we build our pruning method on top of  [SparseGPT](https://github.com/IST-DASLab/sparsegpt). To only use our proposed layer-wise pruning technique to prune the model, please use the following command to prune and tune the model:

``` bash
bash ./scripts/layer_wise_pruning.sh
```

### Layerwise Unified Compressed Model
To test the model performance with only the layer-wise unified compression, please use the following command to compress and tune the model:

``` bash
bash ./scripts/layer_wise_pruning_quantization.sh
```

## Citation
```bibtex
@article{edge_llm,
  title={Edge-LLM:  Enabling Efficient Large Language Model Adaptation on Edge Devices via Layerwise Unified Compression and Adaptive Layer Tuning & Voting},
  author={Zhongzhi Yu, Zheng Wang, Yuhan Li, Haoran You, Ruijie Gao, Xiaoya Zhou, Sreenidhi Reedy Bommu, Yang (Katie) Zhao, Yingyan (Celine) Lin},
  booktitle={61st ACM/IEEE Design Automation Conference (DAC ’24)},
  year={2024}
}
```

Edge-LLM

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README