jobserver

KO | EN

gitlite — search

#hacktoberfest #react #ai #typescript #android #machine-learning #rust #deep-learning #macos #java #windows #vue #awesome-list #open-source

(★ 9)

A nestable Jobserver with thread-safe futures, callbacks, and complete type-hinting

jobserver Latest Version Download

Download Latest Version (.zip)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

Jobserver
=========

A nestable Jobserver with thread-safe futures, callbacks, and type-hinting

Quick Start
-----------

This package is [available from PyPI](https://pypi.org/project/jobserver/): `pip install jobserver`

Try it interactively, for example with `uv`:

```
$ uv run --with jobserver --with IPython python -m IPython --no-banner
In [1]: from jobserver import Jobserver
In [2]: jobserver = Jobserver()
In [3]: future = jobserver(len, (0, 1, 2))
In [4]: future.result()
Out[4]: 3
```

Purpose
-------

[Jobserver](src/jobserver/_jobserver.py) is similar in spirit to
`multiprocessing.Pool` or `concurrent.futures.Executor` with a few differences:

* First, the implementation choices are based upon the [GNU Make
   Jobserver](https://www.gnu.org/software/make/manual/html_node/POSIX-Jobserver.html).
 * Second, as a result, the Jobserver is "nestable" meaning that resource
   constraints will be shared with work submitted by other work.
 * Third, no background threads are spun up to handle any backing
   queues consequently permitting the implementation to play well with
   more 3rd party libraries.
 * Fourth, Futures are eagerly scanned to quickly reclaim resources.
 * Fifth, Futures can detect when a result pipe closes without a result.
 * Sixth, the user can specify additional work acceptance criteria.
   For example, not launching work unless some amount of RAM is available.
 * Lastly, the API communicates when Exceptions occur within a callback.

In particular, `Jobserver` does not inherit from `concurrent.futures.Executor`
because that `Executor` API fundamentally requires a background thread for
asynchronously issuing `concurrent.futures.Future` callbacks.  `Jobserver`,
eschewing threads, consequently is both somehow less-than and more-than a
standard `Executor`.

In contrast, [JobserverExecutor](src/jobserver/_executor.py) combines
a `Jobserver` with a background thread to provide full
`concurrent.futures.Executor` compatibility.  `JobserverExecutor` is a drop-in
replacement for `concurrent.futures.ProcessPoolExecutor` that aims to provide
more robustness at the expense of slower process launching.

Dependencies
------------

None aside from the Python Standard Library.

Comparison with the Python Standard Library
-------------------------------------------

How `Jobserver` and `JobserverExecutor` compare to the standard library's
[`Pool`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.pool.Pool)
and
[`ProcessPoolExecutor`](https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor):

| Feature                           | `Jobserver` | `JobserverExecutor` | `ProcessPoolExecutor` | `Pool` |
|-----------------------------------|:-----------:|:-------------------:|:---------------------:|:------:|
| Nested work shares the slots      |     yes     |         yes         |          no           |   no   |
| Background thread                 |     no      |         yes         |          yes          |  yes   |
| Cancel pending work               |     no      |         yes         |          yes          |   no   |
| Cancel running work               |     yes     |         no          |          no           |   no   |
| Detects individual lost results   |     yes     |         yes         |       partial\*       |   no   |
| User-defined launch criteria      |     yes     |         yes         |          no           |   no   |
| Lambdas/closures via `fork`       |     yes     |         no          |          no           |   no   |
| Lambdas/closures via `spawn`      |     no      |         no          |          no           |   no   |
| Lambdas/closures via `forkserver` |     no      |         no          |          no           |   no   |

\* Each submission owns its own result pipe, so a pipe that closes without a
result is reported as `LostResult` against exactly that one `Future`.
`ProcessPoolExecutor`, by contrast, cannot tell which submission lost its
worker, so it fails all outstanding futures at once.

Testing
-------

Tested with CPython 3.9, 3.10, 3.11, 3.12, 3.13, and 3.14 with [ci](ci) script.<br>
Implementation passes both PEP 8 (per `ruff`) and type-hinting (per `mypy`).<br>
[![Build Status](https://circleci.com/gh/RhysU/jobserver.svg?style=shield)](https://app.circleci.com/pipelines/github/RhysU/jobserver)

License
-------

This Source Code Form is subject to the terms of the Mozilla Public
License, v. 2.0. If a copy of the MPL was not distributed with this
file, You can obtain one at http://mozilla.org/MPL/2.0/.

Examples
--------

* [ex01_basic.py](examples/ex01_basic.py) - Submitting jobs and collecting results
   via shorthand, keyword args, and `submit()`.
 * [ex02_lifecycle.py](examples/ex02_lifecycle.py) - Polling with `done()`,
   `wait()`, and `result()`, plus `reclaim_resources()` and cleanup.
 * [ex03_nested.py](examples/ex03_nested.py) - Nesting submissions so child work
   shares slot constraints with its parent.
 * [ex04_death.py](examples/ex04_death.py) - Detecting a submission whose result
   pipe closes without a result (e.g. a `SIGKILL`-ed worker) via `LostResult`.
 * [ex05_pause.py](examples/ex05_pause.py) - Pausing and resuming a worker via
   `SIGSTOP`/`SIGCONT` through `Future.wait(signal=...)`.
 * [ex06_sleep.py](examples/ex06_sleep.py) - Gating work acceptance on an
   external condition using `replace_sleep()`.
 * [ex07_callbacks.py](examples/ex07_callbacks.py) - Registering `when_done`
   callbacks and draining errors via `CallbackRaised`.
 * [ex08_environment.py](examples/ex08_environment.py) - Setting and unsetting
   environment variables in child processes via `revise_env()`.
 * [ex09_preexec.py](examples/ex09_preexec.py) - Using `replace_preexec()`
   with a plain callable or context manager factory for entry/exit semantics.
 * [ex10_timeouts.py](examples/ex10_timeouts.py) - Using non-blocking polling,
   finite deadlines, and `Blocked` from `result()` and `submit()`.
 * [ex11_pdeathsig.py](examples/ex11_pdeathsig.py) - On Linux, using
   `replace_preexec()` to call `prctl(PR_SET_PDEATHSIG)` so a child dies when
   its parent does.
 * [ex12_executor.py](examples/ex12_executor.py) - Using `JobserverExecutor` as
   a context manager supporting `map()` and `c.f.Future` cancellation.

jobserver

jobserver Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

jobserver

jobserver Latest Version Download

Today's picks

Was this content helpful?

Recent Feedback

Download README