petals
πΈ Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
File Explorer
Download Latest Version (.zip)- check-style.yaml
- push-docker-image.yaml
- run-tests.yaml
- benchmark_forward.py
- benchmark_inference.py
- benchmark_training.py
- prompt-tuning-personachat.ipynb
- prompt-tuning-sst2.ipynb
- __init__.py
- run_dht.py
- run_prod_server.sh
- run_server.py
- __init__.py
- sequence_info.py
- sequence_manager.py
- spending_policy.py
- __init__.py
- config.py
- from_pretrained.py
- inference_session.py
- lm_head.py
- ptune.py
- remote_forward_backward.py
- remote_generation.py
- remote_sequential.py
- sequential_autograd.py
- __init__.py
- block.py
- config.py
- model.py
- __init__.py
- block.py
- config.py
- model.py
- __init__.py
- block.py
- config.py
- model.py
- speculative_model.py
- __init__.py
- block.py
- config.py
- model.py
- __init__.py
- __init__.py
- backend.py
- block_functions.py
- block_selection.py
- block_utils.py
- from_pretrained.py
- handler.py
- memory_cache.py
- reachability.py
- server.py
- task_pool.py
- task_prioritizer.py
- throughput.py
- __init__.py
- asyncio.py
- auto_config.py
- convert_block.py
- cuda_graphs.py
- dht.py
- disk_cache.py
- hf_auth.py
- logging.py
- misc.py
- packaging.py
- peft.py
- ping.py
- random.py
- version.py
- __init__.py
- constants.py
- data_structures.py
- dht_utils.py
- bootstrap.id
- conftest.py
- server2.id
- test_aux_functions.py
- test_block_exact_match.py
- test_cache.py
- test_chained_calls.py
- test_dtype.py
- test_full_model.py
- test_optimized_layers.py
- test_peft.py
- test_priority_pool.py
- test_remote_sequential.py
- test_sequence_manager.py
- test_server_stats.py
- test_speculative_generation.py
- test_tensor_parallel.py
- test_utils.py
- .gitignore
- Dockerfile
- LICENSE
- pyproject.toml
- README.md
- setup.cfg
# Installation Guide
git clone https://github.com/bigscience-workshop/petals
Downloads the entire project code from GitHub to your computer.
cd petals
Moves into the project folder you just downloaded.
2. Official Install Script
Easy Recommendedpip install git+https://github.com/bigscience-workshop/petals
Installs the package published on PyPI directly β no need to clone the source.
brew install python
Installs the pre-built package via Homebrew β no source build required.
Pulled directly from this repo's README.
3. Docker
Easy- Git Needed to download the project code from GitHub.
- Docker Desktop Needed to build and run containers. Install it and keep it running in the background.
sudo docker run -p 31330:31330 --ipc host --gpus all --volume petals-cache:/cache --rm \
Runs the built image as an actual container.
Pulled directly from this repo's README.
4. Python
Easypip install git+https://github.com/bigscience-workshop/petals
Installs the package published on PyPI directly β no need to clone the source.
python -m petals.cli.run_server meta-llama/Meta-Llama-3.1-405B-Instruct
Runs the Python script (or module).
python -m petals.cli.run_server --port 31330 meta-llama/Meta-Llama-3.1-405B-Instruct
Runs the Python script (or module).
python3 -m pip install git+https://github.com/bigscience-workshop/petals
Installs the Python libraries listed in requirements.txt (or similar).
python3 -m petals.cli.run_server meta-llama/Meta-Llama-3.1-405B-Instruct
Runs the Python script (or module).
Pulled directly from this repo's README.