SimCalKV

KO | EN

gitlite — search

#python #go #android #machine-learning #docker #nodejs #java #awesome #awesome-list #vue #pytorch #mcp #ios #open-source

(★ 13)

[MSN 2025] code for the paper Efficient LLMs Inference via Similarity-Aware KV Cache Merging with Bias Calibration.

SimCalKV Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center">Efficient LLMs Inference via Similarity-Aware KV Cache Merging with Bias Calibration</h1>

Yukai Cheng1, Wenxuan Zhou1, Zhizhou Li1, Zhihao Qu1, Baoliu Ye2 
1Key Laboratory of Water Big Data Technology of Ministry of Water Resources, Hohai University, China 
2State Key Laboratory for Novel Software Technology, Nanjing University, China

---
This repository contains the code for the paper *Efficient LLMs Inference via Similarity-Aware KV Cache Merging with Bias Calibration*.

<img src="analysis/Illustration.png" alt="Project Illustration" width="400">

## Setup
The code runs in the environment specified in the `setup/environment.yml`.
You can create the environment using:

```bash
conda env create -f environment.yml
conda activate simcalkv
```
Another way to create virtual environment:
```bash
python3.10 -m venv simcalkv
source simcalkv/bin/activate

pip install -r requirements.txt
```
Detailed information can be found in `setup`.
## Usage

### Quick Start
`run.txt` contains commands that can be executed locally for testing. **nums_token** refers to the number of tokens in the prefill and decoding phase respectively. The specific value can be set in `llama_eval.py`

Run the inference using the following command. The model path needs to be specified manually, while the dataset, compression ratio and compression method can all be found and selected in the code.
```bash
python llama_eval.py \
    --model_name "meta-llama/Meta-Llama-3.1-8B" \
    --dataset cnn_dailymail \
    --split "test[:100]" \
    --compress_ratio 0.4 \
    --kv_method SimCalKV \
    --output_dir ./llama3.1-8b/results_optimize/nums_token512+64
```
* `--dataset`: Dataset name
* `--split`: Dataset split, e.g. `"validation[:10]"` or `"test[:20]"`
* `--compress_ratio`: Compression ratio
* `--kv_method`: Compression method
* `--output_dir`: Directory to save results

### Evaluation
We take **LLama3.1-8B** as an example and provide two example commands for running. **Detailed parameter configurations can be found in** `task_evaluation/instructions.md`.

Summarization (CNN/DailyMail):

```bash
cd task_evaluation
python llama/llama_eval.py --model_name meta-llama/Meta-Llama-3.1-8B --dataset cnn_dailymail --split "test[:100]" --compress_ratio 0.4 --kv_method SimCalKV --output_dir ./llama3.1-8b/results/nums_token1024+64
```

LongBench (Gov_Report):

```bash
cd task_evaluation
python llama/llama_eval.py --model_name meta-llama/Meta-Llama-3.1-8B --dataset LongBench/gov_report --split "train[:200]" --compress_ratio 0.4 --kv_method PyramidInfer --output_dir ./llama3.1-8b/results/nums_token4096+512
```

### Output

* **CSV**: per-sample results
* **JSON**: summary of memory saving, time, throughput and task metrics

SimCalKV

SimCalKV Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

SimCalKV

SimCalKV Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README