accelerated-model-architectures

KO | EN

gitlite — search

#javascript #hacktoberfest #ai #llm #go #golang #android #rust #linux #nodejs #cli #mcp #openai #open-source

accelerated-model-architectures

(★ 95)

No description available.

accelerated-model-architectures Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# <img src="assets/xma.png" width="90px" height="30px" style="vertical-align: middle;"> (Accelerated Model Architectures)

XMA is a repository comprising of fast kernels for model training.  
We are planning on adding lots of experimental and fun model architectures with support for multiple accelerators like NVIDIA, AMD GPUs, Google TPUs and Amazon Trainiums.

Documentation: https://open-lm-engine.github.io/accelerated-model-architectures/

## layers

| functional | CUDA | MPS | Pallas | NKI | ROCm | Triton |
|-| - | - | - | - | - | - |
| GRU | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| LinearAttention | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| M2RNN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| MoE | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ |
| RNN | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |

## functional

| functional | CUDA | MPS | Pallas | NKI | ROCm | Triton |
|-| - | - | - | - | - | - |
| continuous_count | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ |
| cross_entropy | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| fused_linear_cross_entropy | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| fused_residual_add_rmsnorm | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| gru | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| linear_attention | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| m2rnn | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| p_norm | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| pack_sequence | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ |
| rmsnorm | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| rnn | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| softmax | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ |
| swiglu | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ |
| swiglu_packed | ✅ | ❌ | ❌ | ✅ | ❌ | ✅ |
| unpack_sequence | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ |

# Discord Server
Join the [discord server](https://discord.gg/AFDxmjH5RV) if you are interested in LLM architecture or distributed training/inference research.

accelerated-model-architectures

accelerated-model-architectures Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

accelerated-model-architectures

accelerated-model-architectures Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README