Spike-Driven-Transformer-V3 - GitLite

KO | EN

gitlite — search

#javascript #python #react #llm #android #machine-learning #linux #nodejs #cli #kubernetes #awesome-list #security #ios #open-source

Spike-Driven-Transformer-V3

(★ 116)

Offical implementation of "Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training" (IEEE T-PAMI2025)

파일 탐색기

최종 버전 다운로드 (.zip)

- - codeql.yml
- fig.png
- logo.jpg
- main.png
- - .DS_Store
- - .owners.yml
  - .pre-commit-config-zh-cn.yaml
  - .pre-commit-config.yaml
  - .readthedocs.yml
  - __init__.py
  - CITATION.cff
  - dataset-index.yml
  - LICENSE
  - MANIFEST.in
  - model-index.yml
  - pytest.ini
  - Readme.md
  - requirements.txt
  - setup.cfg
  - setup.py
- - .DS_Store
- - .gitignore
  - 0.1.1
  - CITATION.cff
  - dataset-index.yml
  - LICENSE
  - MANIFEST.in
  - model-index.yml
  - Readme.md
  - requirements.txt
  - setup.cfg
  - setup.py
- .DS_Store
.DS_Store
README.md

# 설치 가이드

1. 코드 내려받기

git clone https://github.com/BICLab/Spike-Driven-Transformer-V3

깃허브에서 프로젝트 코드 전체를 내 컴퓨터로 내려받습니다.

cd Spike-Driven-Transformer-V3

방금 내려받은 프로젝트 폴더 안으로 이동합니다.

2. Python

쉬움 추천

사전 준비물

Git GitHub에서 프로젝트 코드를 내려받으려면 필요합니다.
Python 3 설치 시 'Add Python to PATH' 옵션을 꼭 체크하세요 (Windows).

⚠️ 이 프로젝트는 규모가 큰 저장소라, 이 방법이 실제 핵심 제품이 아니라 내부 하위 패키지를 가리키는 것일 수 있습니다. README 전체를 함께 확인해보세요.

pip install -r SDT_V3/Detection/requirements.txt

requirements.txt 등에 명시된 파이썬 라이브러리를 설치합니다.

python <실행할 파일명>.py # README에서 정확한 실행 파일명을 확인하세요

파이썬 스크립트(또는 모듈)를 실행합니다.

✅ 에러 메시지 없이 실행되고 터미널에 안내 문구가 출력되면 정상입니다.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<p align="center">
<img src="./images/logo.jpg" alt="spikezip_logo" width="250" align="center">
</p>

# Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training

[Man Yao*](https://scholar.google.com/citations?user=eE4vvp0AAAAJ), [Xuerui Qiu*](https://scholar.google.com/citations?user=bMwW4e8AAAAJ&hl=zh-CN), [Tianxiang Hu](), [Jiakui Hu](https://github.com/jkhu29), [Yuhong Chou](https://scholar.google.com/citations?user=8CpWM4cAAAAJ&hl=zh-CN&oi=ao), [Keyu Tian](https://scholar.google.com/citations?user=6FdkbygAAAAJ&hl=zh-CN&oi=ao), [Jianxing Liao](), [Luziwei Leng](), [Bo Xu](), [Guoqi Li](https://scholar.google.com/citations?user=qCfE--MAAAAJ&)

*Equal contribution.

BICLab, Institute of Automation, Chinese Academy of Sciences

This repo is the official implementation of [Scaling Spike-driven Transformer with Efficient Spike Firing Approximation Training ](https://arxiv.org/pdf/2411.16061). It currently concludes codes and models for the following tasks:
> **Base Model ImageNet From Scratch**: See [Train_Base.md](SDT_V3/Classification/Model_Base/Train_Base.md).\
> **Large Model ImageNet Pretrain and Finetune**: See [Train_Large.md](SDT_V3/Classification/Model_Large/Train_Large.md).\
> **Object Detection**: See [Detection.md](SDT_V3/Detection/Readme.md).\
> **Semantic Segmentation**: See [Segementation.md](SDT_V3/Segmentation/Readme.md). \
> **DVS**: See [DVS.md](SDT_V3/DVS/Hardvs/Readme.md).

:rocket:  :rocket:  :rocket: **News**:

- **Dec. 19, 2023**: Release the code for training and testing.

## Abstract
The ambition of brain-inspired Spiking Neural Networks (SNNs) is to become a low-power alternative to traditional Artificial Neural Networks (ANNs). This work addresses two major challenges in realizing this vision: the performance gap between SNNs and ANNs, and the high training costs of SNNs. We identify intrinsic flaws in spiking neurons caused by binary firing mechanisms and propose a Spike Firing Approximation (SFA) method using integer training and spike-driven inference. This optimizes the spike firing pattern of spiking neurons, enhancing efficient training, reducing power consumption, improving performance, enabling easier scaling, and better utilizing neuromorphic chips. We also develop an efficient spike-driven Transformer architecture and a spike-masked autoencoder to prevent performance degradation during SNN scaling. On ImageNet-1k, we achieve state-of-the-art top-1 accuracy of 78.5\%, 79.8\%, 84.0\%, and 86.2\% with models containing 10M, 19M, 83M, and 173M parameters, respectively. For instance, the 10M model outperforms the best existing SNN by 7.2\% on ImageNet, with training time acceleration and inference energy efficiency improved by 4.5x and 3.9x, respectively. We validate the effectiveness and efficiency of the proposed method across various tasks, including object detection, semantic segmentation, and neuromorphic vision tasks. This work enables SNNs to match ANN performance while maintaining the low-power advantage, marking a significant step towards SNNs as a general visual backbone.

![avatar](https://raw.githubusercontent.com/BICLab/Spike-Driven-Transformer-V3/main/images/main.png)

## Results
We address the performance and training consumption gap between SNNs and ANNs. A key contribution is identifying the mechanistic flaw of binary spike firing in spiking neurons. To overcome these limitations, we propose a Spike Firing Approximation (SFA) method. This method is based on integer training and spike-driven inference, aiming to optimize the spike firing pattern of spiking neurons. Our results demonstrate that optimization the spike firing pattern leads to comprehensive improvements in SNNs, including enhanced training efficiency, reduced power consumption, improved performance, easier scalability, and better utilization of neuromorphic chips. Additionally, we develop an efficient spike-driven Transformer architecture and a spike masked autoencoder to prevent performance degradation during SNN scaling. By addressing the training and performance challenges of large-scale SNNs, we pave the way for a new era in neuromorphic computing.

![avatar](https://raw.githubusercontent.com/BICLab/Spike-Driven-Transformer-V3/main/images/fig.png)

## Contact Information

```
@ARTICLE{10848017,
  author={Yao, Man and Qiu, Xuerui and Hu, Tianxiang and Hu, Jiakui and Chou, Yuhong and Tian, Keyu and Liao, Jianxing and Leng, Luziwei and Xu, Bo and Li, Guoqi},
  journal={IEEE Transactions on Pattern Analysis and Machine Intelligence}, 
  title={Scaling Spike-Driven Transformer With Efficient Spike Firing Approximation Training}, 
  year={2025},
  volume={47},
  number={4},
  pages={2973-2990},
  doi={10.1109/TPAMI.2025.3530246}}
```

For help or issues using this git, please submit a GitHub issue.

For other communications related to this git, please contact `manyao@ia.ac.cn` and `qiuxuerui2024@ia.ac.cn`.

## Acknowledgement
The pretraining and finetuning of our project are based on [DeiT](https://github.com/facebookresearch/deit), [MCMAE](https://github.com/Alpha-VL/ConvMAE), [Spark](https://github.com/keyu-tian/SparK). and [MAE](https://github.com/facebookresearch/mae). The object detection and semantic segmentation parts are based on [MMDetection](https://github.com/open-mmlab/mmdetection) and [MMSegmentation](https://github.com/open-mmlab/mmsegmentation) respectively. Thanks for their wonderful work.