dagster-slurm

KO | EN

gitlite — search

#javascript #python #hacktoberfest #ai #llm #go #golang #rust #docker #cli #awesome-list #security #pytorch #ios

(★ 57)

Dagster SLURM integration

dagster-slurm 최신버젼 다운로드

최종 버전 다운로드 (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# dagster-slurm

[![DOI](https://camo.githubusercontent.com/6088cff6a8f948c7c271f5658fbb66f95c571bc67bf39df51043848cd3547f7e/68747470733a2f2f6a6f73732e7468656f6a2e6f72672f7061706572732f31302e32313130352f6a6f73732e30393739352f7374617475732e737667)](https://doi.org/10.21105/joss.09795)
[![image](https://img.shields.io/pypi/v/dagster-slurm.svg)](https://pypi.python.org/pypi/dagster-slurm)
[![image](https://img.shields.io/pypi/l/dagster-slurm.svg)](https://pypi.python.org/pypi/dagster-slurm)
[![image](https://img.shields.io/pypi/pyversions/dagster-slurm.svg)](https://pypi.python.org/pypi/dagster-slurm)
[![Ruff](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)

---

Integration for running Dagster assets on Slurm HPC clusters.

`dagster-slurm` lets you take the same Dagster assets from a laptop to a Slurm-backed supercomputer with minimal configuration changes.

**An European sovereign GPU cloud does not come out of nowhere
maybe this project can support making HPC systems more accessible**.

See the [docs](https://dagster-slurm.geoheil.com/) for instructions.

> **Status:** Local execution and one-job-per-asset Slurm runs are production-ready today. Ray workloads can also opt into a run-scoped Slurm allocation when several compatible assets should share one Ray cluster. Heterogeneous jobs remain experimental.

## features

Hydrating supercomputers with Dagster and Slurm to enhance developer experience.

![](https://raw.githubusercontent.com/ascii-supply-networks/dagster-slurm/main/docs/static/img/featured.png)

- **Unified API**: Same asset code works in dev (local) for rapid prototyping but just as well on a massive HPC system via SLURM.
- **Job launch**: One Slurm submission per Dagster asset keeps behaviour predictable by default. Ray assets can explicitly opt into one Slurm allocation per Dagster run when the selected assets share one allocation shape.
- **Pluggable Launchers**: Bash, Ray, Spark (WIP) - easy to add more
- **Environment Packaging**: Automatic `pixi`-based environment packaging for remote execution via `pixi-pack`
- **Connection Pooling**: SSH ControlMaster for efficient remote operations
- **Metrics Collection**: Automatic collection of Slurm job metrics (CPU efficiency, memory, node-hours)
- **Production-Prepared**: Proper error handling, cleanup, health checks

### 📊 Metrics

Automatic metrics collection for all jobs:

- Node-hours consumed
- CPU efficiency
- Max memory usage
- Elapsed time

Visible in Dagster UI metadata.

![Screenshot comparing multiple Dagster runs](https://raw.githubusercontent.com/ascii-supply-networks/dagster-slurm/main/docs/static/img/process_data_run_view.png)

### 🏗️ Architecture

#### Components

- Your server coordinating HPC and non HPC jobs
- The HPC compute resource

![](https://raw.githubusercontent.com/ascii-supply-networks/dagster-slurm/main/docs/static/img/arch-detail-dark.svg)

#### Layers of dagster-slurm

```
Asset Layer (user code)
  ↓
ComputeResource (facade)
  ↓
PipesClient (orchestration)
  ↓
Launcher (what to run) + SSH Pool (how to run)
  ↓
Slurm Execution
```

## Example

See [examples](./examples/) for a small tutorial which starts a slurm cluster in docker.
To use your own full-blown HPC deployment you would have to adapt the SSH connection configuration accordingly.

## Resources

- 2025-11 introduction blog post https://georgheiler.com/post/dagster-slurm/
- 2025-10 hackathon slides https://dagster-slurm.geoheil.com/docs/slides

## Supporters

This work was kindly supported by:

- [ASCII](https://ascii.ac.at/)
- [CSH](https://www.csh.ac.at/)
- [EUROCC AI Hackathon](https://www.openhackathons.org/s/siteevent/a0CUP000013Tp8f2AC/se000375): This work was completed in part at the EUROCC AI Hackathon, part of the Open Hackathons program. The authors would like to acknowledge OpenACC-Standard.org for their support.
- [Austrian Scientific Computing (ASC)](https://asc.ac.at/home/)

dagster-slurm

dagster-slurm 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

dagster-slurm

dagster-slurm 최신버젼 다운로드

오늘의 추천 프로젝트

Was this content helpful?

Recent Feedback

Download README