runai-model-streamer

KO | EN

gitlite — search

#hacktoberfest #llm #machine-learning #rust #deep-learning #docker #macos #windows #awesome #vue #security #ios #mcp #open-source

runai-model-streamer

(★ 340)

No description available.

File Explorer

Download Latest Version (.zip)

- aarch64-linux-gnu.cmake
- devcontainer.json
- Dockerfile
- x86_64-linux-gnu.cmake
- - build-devcontainer.yml
  - on-pr.yaml
  - on-release.yaml
- - azure.cc
  - azure.h
  - azure.ldscript
  - BUILD
- - BUILD
  - gcs.cc
  - gcs.h
  - gcs.ldscript
- - BUILD
  - streamer-mock.cc
- - BUILD
  - s3.cc
  - s3.h
  - s3.ldscript
- - BUILD
  - streamer.cc
  - streamer.h
  - streamer.ldscript
  - streamer_s3_test.cc
  - streamer_test.cc
- - aws.BUILD
  - aws.bzl
  - azure.BUILD
  - azure.bzl
  - BUILD
  - gcp_repo.bzl
- - BUILD
  - configure.bzl
  - deps.bzl
  - README.md
  - rules.bzl
- .bazelrc
- .bazelversion
- .gitignore
- BUILD
- CPPLINT.cfg
- Makefile
- README.md
- rules.bzl
- WORKSPACE
- - benchmarks.md
  - env-vars.md
  - Figure1.png
  - Figure2.png
  - Figure3.png
  - Figure4.png
  - Figure5.png
  - Figure6.png
  - Figure7.png
  - installation.md
  - usage.md
- README.md
- stream_chunks_from_file.ipynb
- stream_multi_safetensors_to_gpu.ipynb
- stream_safetensors_from_custom_object_store.ipynb
- stream_safetensors_from_gcs.ipynb
- stream_safetensors_from_s3.ipynb
- stream_safetensors_to_cpu.ipynb
- stream_safetensors_to_gpu.ipynb
- - LICENSE
  - Makefile
  - requirements.dev
  - run_dist_tests.py
  - setup.py
- - libstreamerazure.so
  - LICENSE
  - Makefile
  - setup.py
- - libstreamergcs.so
  - LICENSE
  - Makefile
  - setup.py
- - libstreamers3.so
  - LICENSE
  - Makefile
  - setup.py
- .gitignore
- Makefile
- - __init__.py
  - test_azure.py
  - test_azurecache.py
- - __init__.py
  - interface.py
  - testcases.py
- - __init__.py
  - test_file_streamer.py
  - test_recovery.py
  - test_safetensors_streamer.py
- - __init__.py
  - test_gcs.py
- - __init__.py
  - test_s3.py
- - __init__.py
  - comparison.py
  - generator.py
- __init__.py
- Makefile
.gitignore
LICENSE
Makefile
README.md
SECURITY.md

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Run:ai Model Streamer
## Overview
The Run:ai Model Streamer is a Python SDK designed to facilitate the streaming of tensors from tensors files to GPU memory with concurrency and streaming. It provides an API for loading SafeTensors files and building AI models, allowing loading models seamlessly.

For documentation click [here](docs/README.md)

For benchmarks click [here](docs/src/benchmarks.md)

## Usage
Install the package
```
pip install runai-model-streamer
```

And stream tensors
```
from runai_model_streamer import SafetensorsStreamer

file_path = "model.safetensors"

with SafetensorsStreamer() as streamer:
    streamer.stream_file(file_path)
    for name, tensor in streamer.get_tensors():
        gpu_tensor = tensor.to('CUDA:0')
```

## Development

Our repository is built using devcontainer ([Further reading](https://containers.dev/))

The following commands should run inside the dev container

> [!NOTE]
> You can use devcontainer-cli tool ([Further reading](https://github.com/devcontainers/cli)) by installing it, and run every command with the following prefix `devcontainer exec --workspace-folder . [COMMAND]`

**Build**
```
make build
```

> [!NOTE]
> We build `libstreamers3.so` and statically link it to libssl, libcrypto, and libcurl. if you would like to use your system libraries by dynamic link to them, run `USE_SYSTEM_LIBS=1 make build`

> [!NOTE]
> On successful build, the `.whl` file would be at `py/runai_model_streamer/dist/<PACKAGE_FILE>` and `py/runai_model_streamer_s3/dist/<PACKAGE_FILE>`

**Run tests**
```
make test
```

**Install locally**
```
pip3 install py/runai_model_streamer py/runai_model_streamer_s3
```

> [!IMPORTANT]
> In order to the CPP to run, you need to install libcurl4 and libssl1.1_1

runai-model-streamer

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

runai-model-streamer

File Explorer

Today's picks

Was this content helpful?

Recent Feedback

Download README