aarambh-studio

β˜… 71 Open GitHub β†—

πŸ¦€ Decoder-only LLM built from scratch in pure Rust using Candle β€” no Python, no PyTorch. Gated DeltaNet + sparse attention, fine-grained MoE, native video/document understanding, long-horizon tool agents, quantization-aware training. Scales: Tiny (25M) to Large (1.3B).

Explore Similar Repositories

LLaMA-MoE-v2 : πŸš€ LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training
SViTE : [NeurIPS'21] "Chasing Sparsity in Vision Transformers: An End-to-End Exploration" by Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, Zhangyang Wang
tgan2 : The official implementation of "Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN"
LLM : LightningLM 0.1V β€” Reference training pipeline for the LightningLM family. 2B dense seed β†’ 5B MoE β†’ 9B MoE β†’ 120B sparse MoE through TurboQuant-PreTraining on a single eight-GPU node. Companion code for *Reversible Foundations*.
BatchTopK : Implementation of the BatchTopK activation function for training sparse autoencoders (SAEs)
// repository documentation