spine

KO | EN

gitlite — search

#javascript #typescript #llm #go #golang #android #linux #docker #java #nodejs #awesome #pytorch #openai #open-source

(★ 9)

Scalable Particle Imaging with Neural Embeddings

spine Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

[![CI](https://github.com/DeepLearnPhysics/spine/actions/workflows/ci.yml/badge.svg)](https://github.com/DeepLearnPhysics/spine/actions/workflows/ci.yml)
[![codecov](https://codecov.io/gh/DeepLearnPhysics/spine/branch/main/graph/badge.svg)](https://codecov.io/gh/DeepLearnPhysics/spine)
[![Documentation Status](https://readthedocs.org/projects/spine/badge/?version=latest)](https://spine.readthedocs.io/latest/)
[![PyPI version](https://badge.fury.io/py/spine.svg)](https://badge.fury.io/py/spine)
[![Python version](https://img.shields.io/pypi/pyversions/spine.svg)](https://pypi.org/project/spine/)

The Scalable Particle Imaging with Neural Embeddings (SPINE) package leverages state-of-the-art Machine Learning (ML) algorithms -- in particular Deep Neural Networks (DNNs) -- to reconstruct particle imaging detector data. This package was primarily developed for Liquid Argon Time-Projection Chamber (LArTPC) data and relies on Convolutional Neural Networks (CNNs) for pixel-level feature extraction and Graph Neural Networks (GNNs) for superstructure formation. The schematic below breaks down the full end-to-end reconstruction flow.

For full SPINE workflows, the recommended runtime is the published SPINE container image released alongside each SPINE version. Use the release-tagged image `ghcr.io/deeplearnphysics/spine:<release>` when reproducibility matters. When in doubt, use `ghcr.io/deeplearnphysics/spine:latest` or omit the tag entirely, which is equivalent in Docker-style image references. Docker is the most direct path on workstations and servers; Apptainer/Singularity is the preferred path on HPC systems that do not allow Docker. A local `pip` installation is mainly intended for post-processing, analysis, visualization, docs, or lightweight development.

![Full chain](https://raw.githubusercontent.com/DeepLearnPhysics/spine/main/docs/source/_static/img/spine-chain-alpha.png)

## Installation

SPINE supports both container-based and local Python installation workflows, but they are not equivalent.

### Recommended Runtime: Released SPINE Container

Every SPINE release publishes a matching container image to GHCR. For end-to-end reconstruction, training, and inference, use a release tag when you want a pinned environment. When in doubt, use `latest` or omit the tag entirely:

```bash
# Equivalent to: docker pull ghcr.io/deeplearnphysics/spine:latest
docker pull ghcr.io/deeplearnphysics/spine

# Example: replace <release> with a SPINE release tag such as 1.2.3
docker pull ghcr.io/deeplearnphysics/spine:<release>
```

Omitting the tag is equivalent to using `latest` in Docker-style image references.

### Docker Path

Use Docker when you have a local workstation or server with container runtime support:

```bash
docker run --gpus all -v $(pwd):/workspace \
    ghcr.io/deeplearnphysics/spine:latest \
    spine --config /workspace/config/uresnet/uresnet_train.yaml --source /workspace/data.root

# Or pin to a specific release
docker run --gpus all -v $(pwd):/workspace \
    ghcr.io/deeplearnphysics/spine:<release> \
    spine --config /workspace/config/uresnet/uresnet_train.yaml --source /workspace/data.root
```

On Apple Silicon macOS systems, the published SPINE image should still be run
as `linux/amd64`. Specify that explicitly by adding
`--platform=linux/amd64` to `docker run`, for example:

```bash
docker run --platform=linux/amd64 --gpus all -v $(pwd):/workspace \
    ghcr.io/deeplearnphysics/spine:<release> \
    spine --config /workspace/config/uresnet/uresnet_train.yaml --source /workspace/data.h5
```

For Jupyter notebook/lab use specifically, avoid the Docker Desktop
combination of Apple Virtualization Framework **with** Rosetta enabled: the
kernel handshake may stall in that setup even though normal SPINE CLI commands
still run. Apple Virtualization Framework without Rosetta and Docker VMM have
both been verified to work for Jupyter with the published image.

### Apptainer / Singularity Path

Use Apptainer or Singularity on HPC systems that do not allow Docker directly. The recommended path is to pull the same released SPINE image from GHCR:

```bash
apptainer pull spine_latest.sif docker://ghcr.io/deeplearnphysics/spine:latest
apptainer exec --nv spine_latest.sif \
    spine --config /workspace/config/uresnet/uresnet_train.yaml --source /workspace/data.root

# Or pin to a specific release
apptainer pull spine_<release>.sif docker://ghcr.io/deeplearnphysics/spine:<release>
apptainer exec --nv spine_<release>.sif \
    spine --config /workspace/config/uresnet/uresnet_train.yaml --source /workspace/data.root
```

The Docker and Apptainer paths consume the same released image; the difference is only the container runtime.

The published image now includes a canonical fallback setup script at
`/opt/spine/setup.sh`. Normal Docker and Apptainer execution should not require
manual sourcing. Some sites expose the image through CVMFS as an unpacked root
filesystem; in that mode, the site integration should still apply the image
environment automatically. If it does not, diagnose the runtime from inside that
unpacked-image environment with:

```bash
source /opt/spine/setup.sh
/opt/spine/check-env.sh
python -c "import ROOT, larcv, spine"
```

### Local Python Installation

Use a local pip installation when you only need downstream tooling such as post-processing, analysis, visualization, documentation, or light development.

**1. Core Package (minimal dependencies)**
```bash
# Essential dependencies: numpy, scipy, pandas, PyYAML, h5py, numba
pip install spine
```

**2. With Visualization Tools**
```bash
# Adds plotly, matplotlib, seaborn for data visualization
pip install spine[viz]
```

**3. Development Environment**
```bash
# Adds testing, formatting, and documentation tools
pip install spine[dev]
```

**4. Everything (except non-pip dependencies, i.e. ROOT, larcv, MinkowskiEngine, etc.)**
```bash
# All optional dependencies (visualization + development tools)
pip install spine[all]
```

### PyTorch ecosystem

#### Option 1: Released SPINE container (recommended)

The published SPINE image already includes the compatible PyTorch, torch-geometric, MinkowskiEngine, and LArCV stack. Use the release-tagged image through Docker or Apptainer as shown above.

#### Option 2: Manual installation (advanced users)
```bash
# Step 1: Install PyTorch with CUDA
pip install torch --index-url https://download.pytorch.org/whl/cu118

# Step 2: Install ecosystem packages (critical order)
pip install --no-build-isolation torch-scatter torch-cluster torch-geometric MinkowskiEngine

# Step 3: Install SPINE
pip install spine[all]
```

> **Why the container is preferred**: the PyTorch ecosystem (torch, torch-geometric, torch-scatter, torch-cluster, MinkowskiEngine) forms an interdependent group requiring exact version compatibility and complex compilation. The released SPINE container pins that stack for you.

### LArCV2

#### Option 1: Use the released SPINE container (recommended)

LArCV2 is already bundled in the published SPINE image.

#### Option 2: Build from source
```bash
# Clone and build the latest LArCV2
git clone https://github.com/DeepLearnPhysics/larcv2.git
cd larcv2
# Follow build instructions in the repository
```

> **Note**: Avoid conda-forge larcv packages as they may be outdated. Use the released SPINE container or build LArCV2 from the official source.

### Development Installation

For developers who want to work with the source code:
```bash
git clone https://github.com/DeepLearnPhysics/spine.git
cd spine
pip install -e .[dev]
```

#### Quick Development Testing (No Installation)

For rapid development and testing without reinstalling the package:

```bash
# Clone the repository
git clone https://github.com/DeepLearnPhysics/spine.git
cd spine

# Install only the dependencies (not the package itself)
# Or alternatively simple run the commands inside the above container
pip install numpy scipy pandas pyyaml h5py numba psutil

# Run directly from source
python src/spine/bin/run.py --config config/uresnet/uresnet_train.yaml --source /path/to/data.root

# Or make it executable and run directly
chmod +x src/spine/bin/run.py
./src/spine/bin/run.py --config your_config.yaml --source data.root
```

> **💡 Development Tip**: This approach lets you test code changes immediately without reinstalling. Perfect for rapid iteration during development.

To build and test packages locally:
```bash
# Build the package
./build_packages.sh

# Install locally built package
pip install dist/spine-*.whl[all]
```

## Usage

### Command Line Interface

**Option 1: Run from the released container:**

```bash
docker run --gpus all -v $(pwd):/workspace \
    ghcr.io/deeplearnphysics/spine:<release> \
    spine --config /workspace/config/uresnet/uresnet_train.yaml --source /workspace/data.h5
```

**Option 2: After installation, use the `spine` command locally:**

```bash
# Run training/inference/analysis
spine --config config/uresnet/uresnet_train.yaml --source /path/to/data.h5
```

**Option 3: Run directly from source (development):**

```bash
# From the spine repository directory
python src/spine/bin/cli.py --config config/uresnet/uresnet_train.yaml --source /path/to/data.h5
```

### Python API

Basic example:
```python
# Necessary imports
from spine.config import load_config_file
from spine.driver import Driver

# Load configuration file  
cfg_path = 'config/uresnet/uresnet_train.yaml'  # or your config file
cfg = load_config_file(cfg_path)

# Initialize driver class
driver = Driver(cfg)

# Execute model following the configuration regimen
driver.run()
```

* Documentation is available at https://spine.readthedocs.io/latest/.
* Tutorials and examples can be found in the documentation.

### Example Configuration Files

Example configurations are available in the `config` folder:

| Configuration name                              | Model          |
| ------------------------------------------------|----------------|
| `uresnet/uresnet_{train,test}.yaml`              | UResNet alone  |
| `uresnet/bayes/uresnet_bayes_{train,test}.yaml`  | Bayesian UResNet |
| `uresnet/ppn/uresnet_ppn_{train,test}.yaml`      | UResNet + PPN  |
| `image/pid/image_pid_{train,test}.yaml`           | Image PID classification |
| `image/energy/image_energy_{train,test}.yaml`     | Image energy regression |
| `image/pid/image_pid_ancestor_{train,test}.yaml`  | Ancestor-tree PID classification |
| `image/energy/image_energy_ancestor_{train,test}.yaml` | Ancestor-tree energy regression |
| `graph_spice/graph_spice_{train,test}.yaml`      | Graph-SPICE    |
| `grappa_shower/grappa_shower_{train,test}.yaml` | GrapPA for shower fragment clustering |
| `grappa_track/grappa_track_{train,test}.yaml` | GrapPA for track fragment clustering |
| `grappa_inter/grappa_inter_{train,test}.yaml` | GrapPA for interaction clustering |

The maintained model directories provide separate training and inference/test
configurations where both modes are supported.

Key configuration parameters you may want to modify:
* `minibatch_size` - per-process batch size for training/inference
* `weight_prefix` - directory to save model checkpoints
* `log_dir` - directory to save training logs
* `tensorboard` - optional TensorBoard scalar logging configuration
* `epochs` - number of training epochs
* `weight_path` - path to checkpoint to load (optional)
* `train` - training instruction block
* `gpus` - GPU IDs to use (leave empty '' for CPU)

Training instructions live in a top-level `train` block. Configurations using
the historical `base.train` location remain supported with a migration warning.
Validation is optional and runs at every configured checkpoint boundary, before
the checkpoint is written:

```yaml
base:
  epochs: 25.0

train:
  weight_prefix: weights/snapshot
  save_epoch: 1.0
  optimizer:
    name: Adam
    lr: 0.001
  lr_scheduler:
    name: ReduceLROnPlateau
    interval: checkpoint
    monitor: loss
    mode: min
    patience: 2

validation:
  file_keys: /path/to/validation/*.root
  fraction: 1.0
  early_stopping:
    monitor: loss
    mode: min
    patience: 5
    min_delta: 0.0
  best_checkpoint:
    monitor: loss
    mode: min
    min_delta: 0.0
```

Validation inherits the training loader's schema, batching, collation and
worker settings while disabling augmentation and random sampling. Joint and
mixed datasets provide both of their validation sources explicitly:

```yaml
validation:
  sources:
    primary:
      file_keys: /path/to/primary_validation/*.root
    secondary:
      file_keys: /path/to/overlay_validation/*.root
```

Use source names `larcv` and `hdf5` instead for a mixed dataset. Joint
validation retains the overlay distribution but uses repeatable sequential
primary/secondary pairing. Validation scalar outputs are logged with a
`val_` prefix and stored in the checkpoint with early-stopping and
best-checkpoint state. When `best_checkpoint` is enabled, an improving
snapshot is atomically copied to `<weight_prefix>-best.ckpt` with its own
checksum. Set `best_checkpoint.path` to choose another stable destination.

Learning-rate schedulers default to `interval: step`, preserving the
historical update after every optimizer step. `interval: checkpoint` advances
the scheduler after each checkpoint-bound validation and before serializing
its state. An optional `monitor` passes the named validation scalar to metric-
aware schedulers such as `ReduceLROnPlateau`; monitored schedulers therefore
require a validation block.

During distributed training, scalar model/loss outputs are averaged across
ranks before CSV and stdout logging. Rank-specific timings and memory metrics
remain local, while TensorBoard events are emitted by rank zero only.

SPINE checkpoints use a versioned, backward-compatible format. In addition to
model weights and progress, new checkpoints record optimizer and optional
learning-rate-scheduler state, per-rank RNG and loader continuation state, the
complete normalized configuration, resolved training/validation dataset
sources, and a runtime manifest containing the SPINE/Python/PyTorch versions,
creation time, distributed world size and source revision when discoverable.
Every checkpoint is written atomically with an adjacent SHA-256 checksum:

```text
snapshot-1000.ckpt
snapshot-1000.ckpt.sha256
```

When training is configured with one top-level `model.weight_path`, SPINE
automatically restores all available training state. Use `resume: true` to
make complete restoration strict: a checkpoint without optimizer state is
then rejected. The historical `restore_optimizer: true` option remains
supported and has the same strict behavior:

```yaml
model:
  weight_path: weights/snapshot-1000.ckpt

train:
  resume: true
  optimizer:
    name: Adam
    lr: 0.001
```

Set `resume: false` explicitly to use `weight_path` only as parameter
initialization and start a new training process at iteration zero. The same
choices are available as `--resume` and `--no-resume` command-line overrides.
Training accepts exactly one checkpoint; `weight_list` remains an
inference-only facility. During automatic resume, a legacy checkpoint without
optimizer state retains its saved progress, restarts the optimizer and emits
a warning.

Epoch progress is restored independently from the global iteration number.
Changing the batch size while resuming therefore continues from the saved
epoch and recomputes the remaining epoch-based run length using the new
number of batches per epoch. The global step remains monotonic. A batch-size
change preserves the saved sample order when sampler state is available, but
necessarily changes subsequent batch boundaries and emits a warning.

Legacy checkpoints without scheduler, RNG or loader state still load. SPINE
warns when a requested resume must restart missing state. Exact stochastic
continuation requires the same distributed world size. DataLoader worker RNG
and prefetch queues cannot be serialized, so configurations with worker-side
stochastic transforms may reproduce the same sample order without being
bit-for-bit identical. Exact mid-epoch sample order also requires a
checkpointable SPINE sampler; generic third-party samplers are replayed to the
saved cursor on a best-effort basis. Numba and external-library RNG state is
not generally introspectable and is not included.

Checkpoint provenance can be queried without constructing the model:

```python
from spine.model import inspect_checkpoint, verify_checkpoint

assert verify_checkpoint("snapshot-1000.ckpt")
info = inspect_checkpoint("snapshot-1000.ckpt", verify=True)
print(info["manifest"])
print(info["config"])
print(info["datasets"])
```

### Running A Configuration File

Basic usage with the `spine` command:
```bash
# Run training/inference directly
spine --config config/uresnet/uresnet_train.yaml --source /path/to/data.h5

# Or run in background with logging
nohup spine --config config/uresnet/uresnet_train.yaml --source /path/to/data.h5 > log_uresnet.txt 2>&1 &
```

You can load a configuration file into a Python dictionary using:
```python
from spine.config import load_config_file

# Load configuration file with SPINE's config loader
cfg = load_config_file('config/uresnet/uresnet_train.yaml')
```

### Reading a Log

A quick example of how to read a training log, and plot something
```python
import pandas as pd
import matplotlib.pyplot as plt
fname = 'path/to/log.csv'
df = pd.read_csv(fname)

# plot moving average of accuracy over 10 iterations
df.accuracy.rolling(10, min_periods=1).mean().plot()
plt.ylabel("accuracy")
plt.xlabel("iteration")
plt.title("moving average of accuracy")
plt.show()

# list all column names
print(df.columns.values)
```

### TensorBoard Logging

TensorBoard logging is optional and configured under the `base` block.

Use the default TensorBoard directory under `log_dir`:

```yaml
base:
    log_dir: logs/train_run
    tensorboard: true
```

This writes TensorBoard event files under `logs/train_run/tensorboard`.

You can also pass TensorBoard writer options directly:

```yaml
base:
    log_dir: logs/train_run
    tensorboard:
        log_dir: tb
        flush_secs: 5
```

In that case, a relative `log_dir` such as `tb` is resolved relative to the
main log directory, so event files are written under `logs/train_run/tb`.

To inspect the logs:

```bash
tensorboard --logdir logs/train_run/tensorboard
```

or, when using a custom TensorBoard directory:

```bash
tensorboard --logdir logs/train_run/tb
```

SPINE still writes its CSV log alongside TensorBoard output. TensorBoard
requires the `tensorboard` Python package to be installed in the runtime
environment.

### Recording network output or running analysis
Documentation for analysis tools and output formatting is available in the main documentation at https://spine.readthedocs.io/latest/.

## Repository Structure
* `bin` contains utility scripts for data processing
* `config` has example configuration files
* `docs` contains documentation source files  
* `src/spine` contains the main package code
* `test` contains unit tests using pytest

Please consult the documentation for detailed information about each component.

## Testing and Coverage

### Running Tests

The SPINE package includes comprehensive unit tests using pytest:

```bash
# Run all tests
pytest

# Run tests for a specific module
pytest test/test_data/

# Run with verbose output
pytest -v
```

### Checking Test Coverage

Test coverage tracking helps ensure code quality and identify untested areas. Coverage reports are automatically generated in our CI pipeline and uploaded to [Codecov](https://codecov.io/gh/DeepLearnPhysics/spine).

To check coverage locally:

```bash
# Run the coverage script (generates terminal, HTML, and XML reports)
./bin/coverage.sh

# Or run pytest with coverage flags directly
pytest --cov=spine --cov-report=term --cov-report=html

# View the HTML report
open htmlcov/index.html
```

The coverage configuration is defined in `pyproject.toml` under `[tool.coverage.run]` and `[tool.coverage.report]`.

## Contributing

Before you start contributing to the code, please see the [contribution guidelines](CONTRIBUTING.md).

### Adding a new model

The SPINE framework is designed to be extensible. To add a new model:

1. **Data Loading**: Parsers exist for various sparse tensor and particle outputs in `spine.io.core.parse`. If you need fundamentally different data formats, you may need to add new parsers or collation functions.

2. **Model Implementation**: Add your model to the `spine.model` package. Include your model in the factory dictionary in `spine.model.factories` so it can be found by the configuration system.

3. **Configuration**: Create a configuration file in the `config/` folder that specifies your model architecture and training parameters.

Once these steps are complete, you should be able to train your model using the standard SPINE workflow.

spine

spine Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

spine

spine Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README