IntLLaMA

KO | EN

gitlite — search

#python #hacktoberfest #llm #machine-learning #deep-learning #linux #nodejs #macos #java #awesome #cli #pytorch #mcp #ios

(★ 19)

IntLLaMA: A fast and light quantization solution for LLaMA

IntLLaMA Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# IntLLaMA: A fast and light quantization solution for LLaMA

## Introduction
IntLLaMA, a fast and light quantization solution reduces gpu-memory requirement and improve computational efficiency while simultaneously preserving model intelligence. Specifically, IntLLaMA facilitates a quantization-friendly distribution of hidden-states by utilizing Random Centralization to address the asymmetry and mitigate the impact of outliers. Meanwhile, Hessian-weighted Singular Value Decomposition(HSVD) is further proposed to compensate for the performance degradation caused by representing the model weights using low bit-width. Benefits from RandC and HSVD, IntLLaMA quantize the weight into 4 bit-width, hidden-state into 8 bit-width sperately and close to full-precision performance in perplexity and MMLU accuracy.

## Update News
- 2023-07-13: Release the code for LoRA instruct fine-tuing, More information can be found in 
- 2023-07-13: Release a 4w8f ChatGLMv2-6B, which archieve in C-eval and speedup . The more detail can be found in Table1 . 
- 2023-07-12: Release the code for convert a full-precision model to quantized model

## Acknowledgement
IntLLaMA was inspired by several open source projects. We are grateful for these excellent projects and list them as follows:
- GPTQ
- AWQ
- Alpaca-LoRA
- Standard-Alpaca

## License
IntLLaMA is released under the Apache 2.0 license.

IntLLaMA

IntLLaMA Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

IntLLaMA

IntLLaMA Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README