BitMoD-HPCA-25 - GitLite

KO | EN

gitlite — search

#javascript #ai #typescript #llm #android #linux #docker #macos #java #windows #awesome-list #vue #ios #mcp

BitMoD-HPCA-25

(★ 158)

No description available.

파일 탐색기

최종 버전 다운로드 (.zip)

- - entry.py
- - chat_demo.ipynb
  - convert_to_hf.py
  - llava_demo.ipynb
  - README.md
- - example_vis.jpg
  - overview.png
- - benchmark.py
  - demo.py
  - offline-weight-repacker.py
  - README.md
  - split_ckpt.py
  - vlm_demo.py
  - vlm_demo_new.py
- .gitignore
- LICENSE
- pyproject.toml
- README.md
- run_awq.sh
- run_eval_ppl.sh
- - quant_weight.py
  - write_results.py
- .gitignore
- llm_eval_c4.py
- llm_eval_wikitext.py
- README.md
- run_exp.sh
- - __init__.py
  - cacti_config.py
  - cacti_simulation.py
  - mem_instance.py
- - energy_plot.ipynb
  - speedup_plot.ipynb
- .gitignore
- accelerator.py
- llm_shape_profile.py
- pe_array.py
- README.md
- run_shape_profile.sh
- test_ant.py
- test_baseline.py
- test_bitmod.py
- test_olive.py
- - __init__.py
  - base.py
  - evaluator.py
  - metrics.py
  - utils.py
- - int_falcon_layer.py
  - int_llama_layer.py
  - int_opt_layer.py
  - LMClass.py
  - models_utils.py
  - transformation.py
- - __init__.py
  - int_linear.py
  - int_matmul.py
  - omni_norm.py
  - omniquant.py
  - quantizer.py
  - utils.py
- - llama-2-13b-int.sh
  - llama-2-13b-mod.sh
  - llama-2-7b-int.sh
  - llama-2-7b-mod.sh
  - llama-3-8b-int.sh
  - llama-3-8b-mod.sh
- .gitignore
- categories.py
- datautils.py
- generate_act_scale_shift.py
- main.py
- parallel_utils.py
- pyproject.toml
- README.md
- utils.py
- - generate_act_scales.py
- - __init__.py
  - calibration.py
  - fake_quant.py
  - mod.py
  - opt.py
  - ppl_eval.py
  - smooth.py
- .gitignore
- LICENSE
- README.md
- run_experiments.sh
- setup.py
LICENSE
README.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration [\[Paper\]](https://arxiv.org/abs/2411.11745)

BitMoD is an algorithm-hardware co-design framework for LLM acceleration using bit-serial hardware with mixture-of-datatypes. It supports diverse precision and data types with a flexible accuracy-efficiency trade-off.

This repository contains the source code for reproducing the experiments of our HPCA'25 paper "BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration".

## News
- [2024/11] 🔥 BitMoD is accepted to [HPCA 2025](https://hpca-conf.org/2025/) !
- [2024/10] 🔥 We have extended [OmniQuant](https://github.com/OpenGVLab/OmniQuant/tree/main) to support BitMoD datatypes.
- [2024/07] 🔥 We have extended [AWQ](https://github.com/OpenGVLab/OmniQuant) and [SmoothQuant](https://github.com/mit-han-lab/smoothquant) to support BitMoD datatypes.

## Getting Started
Every folder in this repo is used for a separate set of experiments in the BitMoD paper. Please go to each folder and follow its `README` to run different experiments.

## Perplexity Results
| Model  | Precision | Quant Method | WikiText2 PPL | C4 PPL |
| ------ | --------- | ------------ | ------------- | ------ |
|  Llama-2-7B  |fp16   |                | 5.47  | 6.97 |
|  Llama-2-7B  |w4g128 | AWQ   + BitMoD | 5.59 | 7.09 |
|  Llama-2-7B  |w4g128 | OmniQ + BitMoD | 5.56 | 7.06 |
|  Llama-2-7B  |w3g128 | AWQ   + BitMoD | 6.07 | 7.64 |
|  Llama-2-7B  |w3g128 | OmniQ + BitMoD | 5.86 | 7.55 |
|  Llama-2-13B |fp16   |                | 4.88 | 6.47 |
|  Llama-2-13B |w4g128 | AWQ   + BitMoD | 4.96 | 6.55 |
|  Llama-2-13B |w4g128 | OmniQ + BitMoD | 4.95 | 6.55 |
|  Llama-2-13B |w3g128 | AWQ   + BitMoD | 5.27 | 6.88 |
|  Llama-2-13B |w3g128 | OmniQ + BitMoD | 5.17 | 6.84 |
|  Llama-2-70B |fp16   |                | 3.32 | 5.52 |
|  Llama-2-70B |w4g128 | AWQ   + BitMoD | 3.40 | 5.57 |
|  Llama-2-70B |w3g128 | AWQ   + BitMoD | 3.70 | 5.77 |
|  Llama-3-8B  |fp16   |                | 6.14 | 8.88 |
|  Llama-3-8B  |w4g128 | AWQ   + BitMoD | 6.50 | 9.32 |
|  Llama-3-8B  |w4g128 | OmniQ + BitMoD | 6.45 | 9.34 |
|  Llama-3-8B  |w3g128 | AWQ   + BitMoD | 7.79 | 11.07 |
|  Llama-3-8B  |w3g128 | OmniQ + BitMoD | 7.56 | 11.06 |
|  Llama-3-70B |fp16   |                | 2.85 | 6.73 |
|  Llama-3-70B |w4g128 | AWQ   + BitMoD | 3.19 | 6.94 |
|  Llama-3-70B |w3g128 | AWQ   + BitMoD | 4.48 | 7.76 |

## Code Structure
```
Repo Root
|---- SmoothQuant-BitMoD   # Running SmoothQuant with basic INT and our proposed BitMoD data types
|---- AWQ-BitMoD           # Running AWQ with basic INT and our proposed BitMoD data types
|---- OmniQuant-BitMoD     # Running OmniQuant with basic INT and our proposed BitMoD data types
|---- bitmod-quant         # Weight-only quantization with different precision and data types (e.g. INT, FP, BitMoD)
|---- bitmod-sim           # BitMoD accelerator simulator
```

## Citation
```bibtex
@article{chen2025hpca,
  title={{BitMoD}: Bit-serial Mixture-of-Datatype LLM Acceleration},
  author={Yuzong Chen and Ahmed F. AbouElhamayed and Xilai Dai and Yang Wang and Marta Andronic and George A. Constantinides and Mohamed S. Abdelfattah},
  journal={IEEE International Symposium on High-Performance Computer Architecture (HPCA)},
  year={2025}
}
```

-----------------

_This work is subject to a patent application filed by Cornell University._