Tutel

(★ 1,010)

Tutel MoE: Optimized Mixture-of-Experts Library, Support GptOss/DeepSeek/Kimi-K2/Qwen3 using FP8/NVFP4/MXFP4

File Explorer

- Asymmetric-RMS-Norms.svg
- CHECKPOINT.md
- DeepSeek-V3.2.png
- DeepSeekR1-tutel-accel.png
- test_baseline.json
- test_tutel.py
- - __init__.py
  - gather.py
  - scatter.py
- - __init__.py
  - antares_ops.h
  - backend.hpp
  - custom_kernel.cpp
- - __init__.py
  - bandwidth_test.py
  - figure.svg
  - helloworld.py
  - helloworld_amp.py
  - helloworld_custom_expert_sharded.py
  - helloworld_custom_gate_expert.py
  - helloworld_ddp.py
  - helloworld_ddp_tutel.py
  - helloworld_from_scratch.py
  - helloworld_switch.py
  - moe_cifar10.py
  - moe_mnist.py
  - nccl_all_gather_v.py
  - nccl_all_to_all_v.py
  - nccl_allreduce_perf.py
  - oai_request.py
  - oai_request_stream.py
  - README.md
- - __init__.py
  - ffn.py
  - llama_ffn.py
- - __init__.py
  - cosine_top.py
  - top.py
- - __init__.py
  - communicate.py
  - fast_dispatch.py
  - jit_compiler.py
  - losses.py
  - moe_layer.py
  - overlap.py
- - __init__.py
  - gating.py
  - sparse.py
- - __init__.py
  - execl.py
  - run.py
- - __init__.py
- - __init__.py
  - patterns.py
  - solver.py
  - spmdx.py
- __init__.py
- jit.py
- moe.py
- net.py
- system.py
.gitignore
CODE_OF_CONDUCT.md
figure.svg
LICENSE
MANIFEST.in
README.md
SECURITY.md
setup.py
SUPPORT.md

1. Get the code

git clone https://github.com/microsoft/Tutel

Downloads the entire project code from GitHub to your computer.

cd Tutel

Moves into the project folder you just downloaded.

Easy Recommended

Prerequisites

> pip3 install -U "huggingface_hub[cli]" --upgrade

Installs the Python libraries listed in requirements.txt (or similar).

>   python3 -m tutel.examples.oai_request_stream --url '0.0.0.0:8000' --prompt 'Write a Python code of the Quicksort algorithm.'

Runs the Python script (or module).

python3 -m torch.distributed.run --nproc_per_node=8 -m tutel.examples.bandwidth_test --size_mb=256

Runs the Python script (or module).

python3 -m tutel.examples.helloworld --dtype=float32

Runs the Python script (or module).

python3 -m tutel.examples.helloworld --dtype=float32 --use_tensorcore

Runs the Python script (or module).

✅ If it runs without errors and prints output in the terminal, it worked.

Pulled directly from this repo's README.