ssm-state-tuning

KO | EN

gitlite — search

#hacktoberfest #typescript #golang #go #android #machine-learning #rust #linux #nodejs #macos #awesome-list #security #pytorch #openai

ssm-state-tuning

(★ 15)

[ACL 2025 Main] Official Pytorch Implementation for "State-offset Tuning: State-based Parameter-Efficient Fine-Tuning for State Space Models"

ssm-state-tuning 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<h1 align="center"> State-offset Tuning: State-based Parameter-Efficient Fine-Tuning for State Space Models</h1>
<h4 align="center">
 
 <a href="https://scholar.google.com/citations?user=Q-ARWkwAAAAJ&hl=eh" target="_blank">Wonjun Kang</a>1,2*, <a href="https://scholar.google.com/citations?user=G1EpeWYAAAAJ&hl=en" target="_blank">Kevin Galim</a>2*, <a href="https://yzeng58.github.io/zyc_cv/" target="_blank">Yuchen Zeng</a>3*, <a href="https://scholar.google.com/citations?user=XJXKp60AAAAJ&hl=en" target="_blank">Minjae Lee</a>2, <a href="http://cvml.ajou.ac.kr/wiki/index.php/Professor" target="_blank">Hyung Il Koo</a>2,4, <a href="https://ece.snu.ac.kr/en/research-faculty/faculty/fulltime?md=view&profid=p041" target="_blank">Nam Ik Cho</a>1
 
 
 1<a href="https://en.snu.ac.kr/index.html" target="_blank">Seoul National University</a>, 2<a href="https://furiosa.ai/" target="_blank">FuriosaAI</a>, 3<a href="https://www.wisc.edu/" target="_blank">UW-Madison</a>, 4<a href="https://www.ajou.ac.kr/en/index.do" target="_blank">Ajou University</a>
 
 </h4>

**Paper Link**: [arXiv](https://arxiv.org/abs/2503.03499)

State Space Models (SSMs) have emerged as efficient alternatives to Transformers, mitigating their quadratic computational cost. However, the application of Parameter-Efficient Fine-Tuning (PEFT) methods to SSMs remains largely unexplored. In particular, prompt-based methods like Prompt Tuning and Prefix-Tuning, which are widely used in Transformers, do not perform well on SSMs. To address this, we propose **state-based methods** as a superior alternative to prompt-based methods. This new family of methods naturally stems from the architectural characteristics of SSMs. State-based methods adjust state-related features directly instead of depending on external prompts. Furthermore, we introduce a novel state-based PEFT method: **State-offset Tuning**. At every timestep, our method directly affects the state at the current step, leading to more effective adaptation. Through extensive experiments across diverse datasets, we demonstrate the effectiveness of our method.

## Updates

- `2025-05-15` Accepted to ACL 2025 Main.
- `2025-03-05` Code released.

### Setup
- Install dependencies
    ```bash
    # Create env
    conda create -n mamba-ssm python=3.10
    conda activate mamba-ssm

# Install pytorch, e.g.,
    conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c nvidia

pip install "numpy<2"

# Install mamba
    pip install "causal-conv1d==1.2.0.post2"
    cd src/mamba
    pip install -e . --no-build-isolation
    cd -

# Install requirements
    pip install -r requirements.txt
    pip install peft==0.9.0 accelerate --no-deps
    ```

- For Spider evaluation, download [Spider](https://drive.usercontent.google.com/download?id=1403EGqzIDoHMdQF4c9Bkyl7dZLZ5Wt6J&export=download&authuser=1) and extract to "data/xlangai_spider/spider"

### Run
```bash
# train
python run_all.py train.py --device 0 --cfg cfg/final/exps/mamba-*/*/*.yaml
```

## Citation
```tex
@article{kang2025state,
  title={State-offset Tuning: State-based Parameter-Efficient Fine-Tuning for State Space Models},
  author={Kang, Wonjun and Galim, Kevin and Zeng, Yuchen and Lee, Minjae and Koo, Hyung Il and Cho, Nam Ik},
  journal={arXiv preprint arXiv:2503.03499},
  year={2025}
}
```

ssm-state-tuning

ssm-state-tuning 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

ssm-state-tuning

ssm-state-tuning 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README