KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #developer-tools #ai-agent #claude-code #codex #typescript #ai-agents #deepseek-harness #mcp #cli #rust

impulse_init

★ 15 Open GitHub ↗

Convolutional Initialization for Data-Efficient Vision Transformers

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Convolutional Initialization for Data-Efficient Vision Transformers
### [Project Page](https://osiriszjq.github.io/impulse_init) | [Paper](https://arxiv.org/pdf/2401.12511.pdf)
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)

[Jianqiao Zheng](https://github.com/osiriszjq/),
[Xueqian Li](https://lilac-lee.github.io/),
[Simon Lucey](https://www.adelaide.edu.au/directory/simon.lucey)<br>
The University of Adelaide

---
🚀 **New! Explore our updated NeurIPS'25 work [Structured Initialization for Vision Transformers](https://github.com/osiriszjq/structured_initialization) — Released in Dec 2025**

---

This is the official implementation of the paper "Convolutional Initialization for Data-Efficient Vision Transformers", including a modified version of [ConvMixer](https://arxiv.org/abs/2201.09792) and [Simple ViT](https://arxiv.org/abs/2205.01580) on CIFAR-10, CIFAR-100, SVHN and [Tiny ImageNet](http://vision.stanford.edu/teaching/cs231n/reports/2015/pdfs/yle_project.pdf). The code is based on [vision-transformers-cifar10](https://github.com/kentaroy47/vision-transformers-cifar10/tree/main)

#### Illustration of different methods to extend 1D encoding
![Illustration of different methods to extend 1D encoding](https://raw.githubusercontent.com/osiriszjq/impulse_init/main/imgs/ConvMixer-ViT.png)
    
    
## Google Colab
[![Explore Siren in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://github.com/osiriszjq/impulse_init/blob/main/Impulse_Initialization.ipynb)<br>
If you want to try out our new initialization for ViT, check this [Colab](https://github.com/osiriszjq/impulse_init/blob/main/Impulse_Initialization.ipynb) for a quick tour.

## Usage
Modify `convmixer.sh` or `vit_pex.sh` first to change the data path and what experiments you want to run, and then just run
```
bash convmixer.sh
```
or
```
bash vit_pex.sh
```

## Citation
```
@article{zheng2024convolutional,
  title={Convolutional Initialization for Data-Efficient Vision Transformers},
  author={Zheng, Jianqiao and Li, Xueqian and Lucey, Simon},
  journal={arXiv preprint arXiv:2401.12511},
  year={2024}
}
```

impulse_init

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README