KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#typescript #ai-agents #cli #claude-code #codex #developer-tools #ai #dsh-plugin #ai-agent #mcp #deepseek-harness #windows

Awesome-Efficient-ViT

★ 55 Open GitHub ↗

Recent Advances on Efficient Vision Transformers

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Awesome Efficient ViT [![Awesome](https://cdn.rawgit.com/sindresorhus/awesome/d7305f38d29fed78fa85652e3a63e154dd8e8829/media/badge.svg)](https://github.com/sindresorhus/awesome)
A curated list of recent papers on efficient vision transformers.

Transformers belong to the big family of deep learning (deep neural networks). Conceivably, we would see many papers under the umbrella of *efficient transformers* are actually reusing the techniques invented before for *efficient deep learning*. Welcome to check the [paper collection on efficient deep learning](https://github.com/MingSun-Tse/Efficient-Deep-Learning)!

## Surveys
- 2020.09-[Efficient Transformers: A Survey](https://arxiv.org/abs/2009.06732)

## Papers
**2019**
- 2019-NIPS-[Are sixteen heads really better than one?](https://arxiv.org/abs/1905.10650)

**2020**
- 2020.11-[Long Range Arena: A Benchmark for Efficient Transformers](https://arxiv.org/abs/2011.04006)

**2021**
- 2021-KDDw-[Vision Transformer Pruning](https://arxiv.org/abs/2104.08500)
- 2021-TCPS-[TPrune: Efficient transformer pruning for mobile devices](https://dl.acm.org/doi/pdf/10.1145/3446640)
- 2021.05-[MLPruning: A Multilevel Structured Pruning Framework for Transformer-based Model](https://arxiv.org/abs/2105.14636) [[Code](https://github.com/yaozhewei/MLPruning)]
- 2021.07-[Learning Efficient Vision Transformers via Fine-Grained Manifold Distillation]()
- 2021.09-[HFSP: A Hardware-friendly Soft Pruning Framework for Vision Transformers](https://openreview.net/forum?id=dhLChxJwgMR)
- 2021.11-[Pruning Self-attentions into Convolutional Layers in Single Path](https://arxiv.org/abs/2111.11802) [[Code](https://github.com/ziplab/SPViT)]
- 2021.11-[A Memory-saving Training Framework for Transformers](https://arxiv.org/abs/2111.11124) [[Code](https://github.com/ziplab/Mesa)]

**2022**
- 2022-AAAI-[Less is More: Pay Less Attention in Vision Transformers](https://arxiv.org/abs/2105.14217)
- 2022-ICLR-[Unified Visual Transformer Compression](https://openreview.net/forum?id=9jsZiUgkCZP)
- 2022-CVPR-[Patch Slimming for Efficient Vision Transformers](https://arxiv.org/abs/2106.02852)
- 2022-CVPR-[MiniViT: Compressing Vision Transformers with Weight Multiplexing](https://arxiv.org/abs/2204.07154)
- 2022-ECCV-[An Efficient Spatio-Temporal Pyramid Transformer for Action Detection](https://arxiv.org/abs/2207.10448)
- 2022-NIPS-[Fast Vision Transformers with HiLo Attention](https://arxiv.org/abs/2205.13213) [[Code](https://github.com/ziplab/LITv2)]
- 2022-NIPS-[EcoFormer: Energy-Saving Attention with Linear Complexity](https://arxiv.org/abs/2209.09004) [[Code](https://github.com/ziplab/EcoFormer)]
- 2022-NIPS-[EfficientFormer: Vision Transformers at MobileNet Speed](https://arxiv.org/abs/2206.01191) [[Code](https://github.com/snap-research/EfficientFormer)]

## Other Resources:
- [ViT-cifar10-pruning](https://github.com/Cydia2018/ViT-cifar10-pruning)

## Related Repos and Websites
- [Awesome-Visual-Transformer](https://github.com/dk-liang/Awesome-Visual-Transformer)
- [Efficient-Deep-Learning](https://github.com/MingSun-Tse/Efficient-Deep-Learning)
- [Awesome-NAS](https://github.com/D-X-Y/Awesome-NAS)
- [Awesome-Pruning](https://github.com/he-y/Awesome-Pruning)
- [Awesome-Knowledge-Distillation](https://github.com/FLHonker/Awesome-Knowledge-Distillation)
- [MS AI-System open course](https://github.com/microsoft/AI-System/tree/main/Lectures)
- [caffe-int8-convert-tools](https://github.com/BUG1989/caffe-int8-convert-tools)
- [Neural-Networks-on-Silicon](https://github.com/fengbintu/Neural-Networks-on-Silicon)
- [Embedded-Neural-Network](https://github.com/ZhishengWang/Embedded-Neural-Network)
- [model_compression](https://github.com/j-marple-dev/model_compression)
- [model-compression](https://github.com/666DZY666/model-compression) (in Chinese)
- [Efficient-Segmentation-Networks](https://github.com/xiaoyufenfei/Efficient-Segmentation-Networks)
- [AutoML NAS Literature](https://www.automl.org/automl/literature-on-neural-architecture-search/)
- [Papers with code](https://paperswithcode.com/task/network-pruning)
- [ImageNet Benckmark](https://paperswithcode.com/sota/image-classification-on-imagenet)
- [Self-supervised ImageNet Benckmark](https://paperswithcode.com/sota/self-supervised-image-classification-on)
- [NVIDIA Blog with Sparsity Tag](https://developer.nvidia.com/blog/tag/sparsity/)

Awesome-Efficient-ViT

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README