ShallowSpeed

KO | EN

gitlite — search

#react #llm #golang #android #machine-learning #rust #linux #docker #nodejs #windows #cli #vue #mcp #open-source

(★ 166)

Small scale distributed training of sequential deep learning models, built on Numpy and MPI.

파일 탐색기

최종 버전 다운로드 (.zip)

- - PP_pebble_graph.gif
  - title_picture.jpg
- .gitkeep
- DDP_PyTorch_MNIST.py
- __init__.py
- dataset.py
- functional.py
- layers.py
- optimizer.py
- pipe.py
- utils.py
- test_dataset.py
- test_functional.py
- test_layers.py
- test_schedules.py
.gitignore
.pre-commit-config.yaml
download_dataset.py
environment.yml
README.md
setup.py
train.py

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Shallowspeed
![stability-wip](https://img.shields.io/badge/stability-work_in_progress-lightgrey.svg)

A tiny POC implementation of distributed training for sequential deep learning models.
Implemented using plain Numpy & mpi4py.

![](https://raw.githubusercontent.com/siboehm/ShallowSpeed/master/.github/assets/title_picture.jpg)

Currently implements:
- Sequential models / deep MLPs, training using SGD.
- Data parallel training with interleaved communication & computation, similar to PyTorch's [DistributedDataParallel](https://arxiv.org/abs/2006.15704).
- Pipeline parallel training:
  - Naive schedule without interleaved stages.
  - [Gpipe](https://arxiv.org/abs/1811.06965) schedule with interleaved FWD & interleaved BWD.
  - (soon) [PipeDream Flush](https://arxiv.org/abs/2006.09503) schedule with additional inter-FWD & BWD interleaving.
- Any combination of DP & PP algorithms.

## Setup
```bash
conda env create
pip install -e .
# M1 Macs: conda install "libblas=*=*accelerate"
python download_dataset.py
pytest
```

## Usage
```bash
# Sequential training
python train.py
# Data parallel distributed training
mpirun -n 4 python train.py --dp 4
# Pipeline parallel distributed training
mpirun -n 4 python train.py --pp 4 --schedule naive
# Data & pipeline parallel distributed training
mpirun -n 8 python train.py --dp 2 --pp 4 --schedule gpipe
```

## Internals
![](https://raw.githubusercontent.com/siboehm/ShallowSpeed/master/.github/assets/PP_pebble_graph.gif)

ShallowSpeed

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

ShallowSpeed

파일 탐색기

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README