BatchTopK

โ˜… 67 Open GitHub โ†—

Implementation of the BatchTopK activation function for training sparse autoencoders (SAEs)

Explore Similar Repositories

tgan2 : The official implementation of "Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN"
aarambh-studio : ๐Ÿฆ€ Decoder-only LLM built from scratch in pure Rust using Candle โ€” no Python, no PyTorch. Gated DeltaNet + sparse attention, fine-grained MoE, native video/document understanding, long-horizon tool agents, quantization-aware training. Scales: Tiny (25M) to Large (1.3B).
LLM : LightningLM 0.1V โ€” Reference training pipeline for the LightningLM family. 2B dense seed โ†’ 5B MoE โ†’ 9B MoE โ†’ 120B sparse MoE through TurboQuant-PreTraining on a single eight-GPU node. Companion code for *Reversible Foundations*.
Sparse-vLLM : A sparse-first inference engine (sparsevllm). It also contains DeltaKV compressor training + evaluation tooling (deltakv).
2by4-pretrain : Efficient 2:4 sparse training algorithms and implementations
// repository documentation