KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#typescript #ai-agents #developer-tools #cli #codex #claude-code #dsh-plugin #mcp #react #agent #dsh #deepseek-harness

buildomat

★ 69 Open GitHub ↗

a software build labour-saving device

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

<img src="./www/buildomat_wide.png" alt="hammer logo"> 
B U I L D O M A T 
a software build labour-saving device

<hr>

Buildomat manages the provisioning of ephemeral UNIX systems (e.g., instances
in AWS EC2) on which to run software builds.  It logs job output, collects
build artefacts, and reports status.  The system integrates with GitHub through
the Checks API, to allow build jobs to be triggered by pushes and pull
requests.

## Components

Buildomat is made up of a variety of crates, loosely grouped into areas
of related functionality:

```
$ cargo xtask crates
buildomat                    /bin
buildomat-agent              /agent
buildomat-bunyan             /bunyan
buildomat-client             /client
buildomat-common             /common
buildomat-database           /database
buildomat-server             /server
buildomat-types              /types

buildomat-factory-aws        /factory/aws
buildomat-factory-lab        /factory/lab

buildomat-github-common      /github/common
buildomat-github-database    /github/database
buildomat-github-dbtool      /github/dbtool
buildomat-github-ghtool      /github/ghtool
buildomat-github-server      /github/server

xtask                        /xtask
```

### Buildomat Core

The buildomat core is responsible for authenticating users and remote services,
for managing build systems, and for running jobs and collecting output.

#### Server (`buildomat-server`, in `server/`)

The core buildomat API server.  Coordinates the creation, tracking, and
destruction of workers in which to execute jobs.  This component sits at the
centre of the system and is used by the GitHub integration server, the client
command, the agent running within each worker for control of the job, and any
factories.

#### Client Command (`buildomat`, in `bin/`)

A client tool that uses the client library to interface with and manipulate the
core server.  The tool has both administrative and user-level functions,
expressed in a relatively regular hierarchy of commands; e.g., `buildomat job
run` or `buildomat user ls`.

```
$ ./target/release/buildomat
Usage: buildomat [OPTS] COMMAND [ARGS...]

Commands:
    info                get information about server and user account
    control             server control functions
    job                 job management
    user                user management

Options:
        --help          usage information
    -p, --profile PROFILE
                        authentication and server profile

ERROR: choose a command
```

#### Client Library (`buildomat-client`, in `client/`)

A HTTP client library for accessing the core buildomat server.  This client is
generated at build time by
[`progenitor`](https://github.com/oxidecomputer/progenitor), an OpenAPI client
generator.

The client is generated based an OpenAPI document managed in the repository and
generated by Dropshot based on the implementation of the server and then
checked in to the repository.  If you make changes to the API exposed by the
core server, you will need to regenerate the document, `client/openapi.json`,
using:

```
$ cargo xtask openapi
```

#### Agent (`buildomat-agent`, in `agent`/)

A process that is injected into an ephemeral AWS EC2 instance to allow the
buildomat core server to take control and run jobs.  This process receives
single-use credentials at provisioning time from the core server, and connects
out to receive instructions.  The agent does not require a public IP, or any
direct inbound connectivity, to allow agents to run inside remote NAT
environments.

### Factories

Buildomat jobs are specified to execute within a particular target environment.
Concrete instances of those target environments (known as _workers_) are
created, managed, and destroyed by _factories_.  Factories are long-lived
server processes that connect to the core API and offer to construct workers as
needed.  When a worker has finished executing the job, or when requested by an
operator, the factory is also responsible for freeing any resources that were
in use by the worker.

#### AWS Factory (`buildomat-factory-aws` in `factory/aws/`)

The AWS factory creates ephemeral AWS instances that are used to run one job
and are then destroyed.  The factory arranges for the agent to be installed and
start automatically in each instance that is created.  The factory is
responsible for ensuring no stale resources are left behind, and for enforcing
a cap on the concurrent use of resources at AWS.  Each target provided by an
AWS factory can support a different instance type (i.e., CPU and RAM capacity),
a different image (AMI), and a different root disk size.

#### Lab Factory (`buildomat-factory-lab` in `factory/lab/`)

The lab factory uses IPMI to exert control over a set of physical lab systems.
When a worker is required, a lab system is booted from a ramdisk and the agent
is started, just as it would be for an AWS instance.  From that point on,
operation is quite similar to AWS instances: the agent communicates directly
with the core API.  When tearing down a lab worker, the machine is rebooted
(again via IPMI) to clear out the prior ramdisk state.  Each target provided by
a lab factory can boot from a different ramdisk image stored on a local server.

### GitHub Integration (formerly known as Wollongong)

The GitHub-specific portion of the buildomat suite sits in front of the core
buildomat service.  It is responsible for receiving and processing
notifications of new commits and pull requests on GitHub, starting any
configured build jobs, and reporting the results so that they are visible
through the GitHub user interface.

#### Server (`buildomat-github-server`, in `github/server/`)

This server acts as a [GitHub App](https://docs.github.com/en/developers/apps).
It is responsible for processing incoming [GitHub
webhooks](https://docs.github.com/en/developers/webhooks-and-events/webhooks/about-webhooks)
that notify the system about commits and pull requests in authorised
repositories.  In addition to relaying jobs between GitHub and the buildomat
core, this service provides an additional HTML presentation of job state (e.g.,
detailed logs) and access to any artefacts that jobs produce.  This server
keeps state required to manage the interaction with GitHub, but does not store
job data; requests for logs or artefacts are proxied back to the core server.

#### Database Tool (`buildomat-github-dbtool`, in `github/dbtool/`)

This tool can be used to inspect the database state kept by the GitHub
integration as it tracks GitHub pull requests and commits.  Unlike the core
client tool, this program directly interacts with a local SQLite database.

```
$ buildomat-github-dbtool
Usage: buildomat-github-dbtool COMMAND [ARGS...]

Commands:
    delivery (del)      webhook deliveries
    repository (repo)   GitHub repositories
    check               GitHub checks

Options:
    --help              usage information

ERROR: choose a command
```

Of particular note, the tool is useful for inspecting and replaying received
webhook events; e.g.,

```
$ buildomat-github-dbtool del ls
SEQ   ACK RECVTIME             EVENT          ACTION
0     1   2021-10-05T01:58:32Z ping           -
1     1   2021-10-05T02:25:33Z installation   created
2     1   2021-10-05T02:26:53Z push
3     1   2021-10-05T02:26:53Z check_suite    requested
4     1   2021-10-05T02:26:56Z check_suite    completed
5     1   2021-10-05T02:26:56Z check_run      completed
6     1   2021-10-05T02:26:56Z check_run      created
7     1   2021-10-05T02:26:56Z check_run      created
8     1   2021-10-05T02:26:57Z check_run      created
...
```

The `buildomat-github-dbtool del unack SEQ` command can be used to trigger the
reprocessing of an invididual webhook message.

## Per-repository Configuration

Buildomat works as a [GitHub App](https://docs.github.com/en/developers/apps),
which is generally "installed" at the level of an
[Organisation](https://docs.github.com/en/organizations).  Installing the App
allows buildomat to receive notifications about events, such as git pushes and
pull requests, from all repositories (public and private) within the
organisation.  In order to avoid accidents, buildomat requires that the service
be explicitly configured for a repository before it will take any actions.

Per-repository configuration is achieved by creating a file **in the default
branch** (e.g., `main`) of the repository in question, named
`.github/buildomat/config.toml`.  This file is written in
[TOML](https://toml.io/), with a handful of simple values.  Supported
properties in this file include:

- `enable` **(boolean)**

Must be present and have the value `true` in order for buildomat to consider
  the repository for jobs; e.g.,

```toml
  enable = true
  ```

- `org_only` **(boolean, defaults to `true` if missing)**

If set to `true`, or missing from the file, buildomat will not automatically
  run jobs in response to pull requests opened by users that are not a member
  of the GitHub Organisation which owns the repository.  If set to `false`, any
  GitHub user can cause a job to be executed.

This property is important for security if your repository is able to create
  any jobs that have access to secrets, or to restricted networks.

- `allow_users` **(array of strings, each a GitHub login name)**

If specified, jobs will be started automatically for users in this list,
  regardless of whether they are a member of the Organisation that owns the
  repository or not, and regardless of the value of the `org_only` property.

This is often useful for pre-authorising jobs driven by Pull Requests
  made by various automated systems; e.g.,

```toml
  allow_users = [
          "dependabot[bot]",
          "renovate[bot]",
  ]
  ```

Note that buildomat will only ever read this configuration file from the most
recent commit in the default branch of the repository, not from the contents of
another branch or pull request.  This is of particular importance for
security-sensitive properties like `org_only`, where the policy set by users
with full write access to the repository must not be overridden by changes from
potentially untrusted users.  If a pull request with a malicious policy change
is merged, it will then be in the default branch and active for subsequent pull
requests; maintainers must carefully review pull requests that change this
file.

## Specifying Jobs

Once you have configured buildomat at the repository level, you can specify
some number of jobs to execute automatically in response to pushes and pull
requests.  While per-repository configuration is read from the default branch,
jobs are read from the commit under test.

Jobs are specified as `bash` programs with some configuration directives
embedded in comments.  These job files must be named
`.github/buildomat/jobs/*.sh`.  Unexpected additional files in
`.github/buildomat/jobs` will result in an error.

Job files should begin with an interpreter line, followed by TOML-formatted
configuration prefixed with `#:` so that they will be identified as
configuration by buildomat, but ignored by the shell.  For example, a minimal
job that would just execute `uname -a`:

```bash
#!/bin/bash
#:
#: name = "build"
#: variety = "basic"
#:
uname -a
```

The minimum set of properties that must always appear in the TOML frontmatter
is:

- `name` **(string)**

Must be present in all jobs.  This name is used for at least two things: as
  the name of the Check Run in the GitHub user interface, and when specifying
  that some other job depends on this job.  The job name must be unique amongst
  all jobs within the commit under test.

In general, it is probably best to keep these short, lower-case, and without
  spaces.  It is conventional to use the same name for the job file and the
  job, e.g., `name = "build"` in file `.github/buildomat/jobs/build.sh`.

- `variety` **(string)**

To allow the system to evolve over time, a job must specify a _variety_,
  which defines some things about the way a job executes and what additional
  configuration options are required or available.

These properties are optional, but not variety-specific:

- `enable` **(boolean)**

To exclude a particular job file from processing, set this to `false`.  If
  not specified, this property defaults to `true`.  This allows a job to be
  temporarily disabled without needing to be removed from the repository.

The rest of the configuration is variety-specific.

### Variety: Basic

Each **basic** variety job (selected by specifying `variety = "basic"` in the
frontmatter) takes a single `bash` program and runs it in an ephemeral
environment.  The composition of that environment, such as compute and memory
capacity or the availability of specific toolchains and other software, depends
on the `target` option.

Basic variety jobs can produce output files (see the configuration options
`output_rules` and `publish`).  They can also depend on the successful
completion of other jobs, gaining access to any output files from the upstream
job (see the `dependencies` option).  Jobs are generally executed in parallel,
unless they are waiting for a dependency or for capacity to become available.

#### Execution Environment

By default, an ephemeral system (generally a virtual machine) will be
provisioned for each job.  The system will be discarded at the end of the job,
so no detritus is left behind.  Once the environment is provisioned, the `bash`
program in the job file is executed as-is.

Jobs are executed as an unprivileged user, `build`, with home directory
`/home/build`.  If required, this user is able to escalate to `root` privileges
through the use of [pfexec(1)](https://illumos.org/man/1/pfexec).  Systems that
do not have a native `pfexec` will be furnished with a compatible wrapper
around a native escalation facility, to ease the construction of cross-platform
jobs.

By default, the working directory for the job is based on the name of the
repository; e.g., for https://github.com/oxidecomputer/buildomat, the working
directory would be `/work/oxidecomputer/buildomat`.  The system will arrange
for the repository to be cloned at that location with the commit under test
checked out.  A simple job could directly invoke some build tool like `gmake`
or `cargo build`, and the build would occur at the root of the clone.  The
`skip_clone` configuration option can disable this behaviour.

Most targets provide toolchains from common metapackages like
`build-essential`; e.g., `gmake` and `gcc`.  If a Rust toolchain is required,
one can be requested through the `rust_toolchain` configuration option.  This
will be installed using [rustup](https://rustup.rs/).

##### Environment Variables

While the complete set of environment variables is generally target-specific,
the common minimum for all targets includes:

- `BUILDOMAT_JOB_ID` will be set to the unique ID of this job
- `CI` will be set to `true`
- `GITHUB_REPOSITORY` set to `owner/repository`; e.g., `oxidecomputer/buildomat`
- `GITHUB_SHA` set to the commit ID of the commit under test
- If the commit under test is part of a branch, then `GITHUB_BRANCH` will be
  set to the branch name (e.g., `main`) and `GITHUB_REF` will be set to the ref
  name; e.g., `res/heads/main`.
- `HOME`, set to the home directory of the build user
- `USER` and `LOGNAME`, set to the username of the build user
- `PATH` set to include relevant directories for toolchains and other
  useful software
- `TZ` will be set to `UTC`
- `LANG` and `LC_ALL` will be set to `en_US.UTF-8`

##### Available Commands

Cross-platform shell programming can be challenging due to differences between
different operating systems.  To make this a little easier, we ensure that each
buildomat target can provide a basic suite of tools that are helpful in
constructing succint jobs:

- [pfexec(1)](https://illumos.org/man/1/pfexec) allows escalation from the
  unprivileged build user to `root`; e.g., `pfexec id -a`.
- [ptime(1)](https://illumos.org/man/1/ptime) runs a program and provides
  (with `-m`, detailed) timing information; e.g., `ptime -m cargo test`.
- [banner(1)](https://illumos.org/man/1/banner) prints its arguments in
  large letters on standard output, and is useful for producing headings
  in job log output; e.g., `banner testing`.

#### Configuration

Configuration properties supported for basic jobs include:

- `access_repos` **(array of strings)**

Jobs can be created in both public and private repositories.  Public
  repositories are available to everybody, but private repositories require
  appropriate credentials.  By default, an ephemeral, read-only token is
  injected into the execution environment (in the `$HOME/.netrc` file) that is
  able to access only the repository directly under test.

If a job requires access to additional private repositories beyond the
  direct repository, they may be specified in this list, in the form
  `owner/repository`; e.g.,

```bash
  #: access_repos = [
  #:	"oxidecomputer/clandestine-project",
  #:	"oxidecomputer/secret-plans",
  #: ]
  ```

Note that this option only works for repositories within the same
  organisation as the direct repository.  Using the option will trigger a
  requirement for job-level authorisation by a member of the organisation.

- `dependencies` **(table)**

A job may depend on the successful completion of one or more other jobs from
  the same commit under test.  If the dependency is cancelled or fails to
  complete successfully for some other reason, that failure will be propagated
  forward as a failure of this job.

Each entry in the dependencies table is itself a table with a name for the
  dependency, and the following per-dependency properties:

* `job` **(string)**

Specifies the job that this job should wait on for execution.  The `job`
    value must exactly match the `name` property of some other `basic` variety
    job available in the same commit.

Any artefacts output by the job named in the dependency will be made
  available automatically under `/input/$dependency` using the dependency
  name.  For example, consider this dependency directive:

```bash
  #: [dependencies.otherjob]
  #: job = "the-other-job!"
  ```

If the job with the name `the-other-job!` produces an output file,
  `/tmp/output.zip`, then it will be made available within this job as the file
  `/input/otherjob/tmp/output.zip`.

Using this facility, one can easily split a job into a "build" phase that
  runs under a target with access to toolchains, and one or more "test" phases
  that can take the build output and run it in under another target that might
  not have a toolchain or may have access to other resources that have limited
  availability like test hardware.

Jobs can also depend on more than one other job, allowing a job to aggregate
  artefacts from several other jobs together in one place.  This might be
  useful when building binaries for more than one different OS, with a final
  step that publishes multi-OS packages if all the other builds were
  successful.

Cycles in the dependency graph are not allowed.

- `output_rules` **(array of strings)**

Jobs may produce artefacts that we wish to survive beyond the lifetime of the
  ephemeral build environment.  A job may specify one or more files for
  preservation by the system; e.g., a build job may produce binaries or
  packages that can then be downloaded and installed, or a test job may produce
  JUnit XML files or other diagnostic logs that can be inspected by engineers.

The `output_rules` property is a list of `/`-anchored glob patterns that
  match files in the ephemeral machine; e.g., `/tmp/*.txt` would match
  `/tmp/something.txt` but not `/tmp/nothing.png`.  Like the shell, a single
  asterisk (`*`) will not descend into a hierarchy of directories.  If you want
  to match recursively, a double asterisk (`**`) pattern will match the current
  directory or any directory under that directory, but not files.  You can
  combine these to get a recursive match; e.g., `/tmp/**/*.txt` would match
  `/tmp/a.txt`, `/tmp/dir/a.txt`, and `/tmp/dir/dir/a.txt`.

By default, it is not an error to specify a pattern that does not match any
  files.  Provided the job is not cancelled, matching files are uploaded
  whether the job program exits with a zero status (denoting success) or a
  non-zero status (denoting failure).  These behaviours can be used to upload
  diagnostic logs left behind by unexpected test failures that are cleaned up
  on success; e.g.,

```bash
  #: output_rules = [
  #:	"/tmp/test_output/*",
  #: ]
  ```

If the success of a job _requires_ that a particular artefact is produced,
  the `=` prefix may be used to signify "this rule must match at least one
  file".  If the rule does not match at least one output file, the job is
  marked as failed even if the job program otherwise succeeded.  This can be
  used to make sure that, say, a release binary build job produces an archive
  with the expected name; e.g.,

```bash
  #: output_rules = [
  #:	"=/work/pkg/important.tar.gz",
  #:	"=/work/pkg/important.sha256.txt",
  #: ]
  ```

By default, the system attempts to ensure that a job has not accidentally
  left background processes running that continue to modify the output
  artefacts.  If the size or modified time of a file changes while it is being
  uploaded, the job will fail.  To relax this restriction, the `%` prefix may
  be used to signify that "this file is allowed to change while it is being
  uploaded".  The `%` prefix will also ignore a file that is completely removed
  by a background process before it is able to be uploaded.  This is used to
  make best effort uploads of diagnostic log files for background processes
  which may continue running even though the job is nominally complete; e.g.,

```bash
  #: output_rules = [
  #:	"%/var/svc/log/*.log",
  #: ]
  ```

To exclude specific files from upload, the `!` prefix can be used to signify
  that "any file that matches this pattern should be ignored, even if it was
  nominally included by another pattern".  Order in the array is not important;
  a match of any exclusion rule will prevent that file from behing uploaded.
  For example, to upload anything left in `/tmp` except for pictures:

```bash
  #: output_rules = [
  #:	"/tmp/*",
  #:	"!/tmp/*.jpg",
  #: ]
  ```

The must-match (`=`) and allow-change (`%`) prefixes may be combined in a
  single output rule.  The exclusion prefix (`!`) may not be combined with any
  other prefix.  For example, to require at least one log file (which may still
  be growing) that is not `big-and-useless.log`:

```bash
  #: output_rules = [
  #:	"=%/tmp/*.log",
  #:	"!/tmp/big-and-useless.log",
  #: ]
  ```

- `publish` **(array of tables)**

Some jobs may wish to publish a specific subset of their output artefacts at
  a predictable URL based on the commit ID of the commit under test, for
  reference by other jobs from other repositories, or end user tools.

Each table in the `publish` array of tables must contain these properties:

* `from_output` **(string)**

Specify the full path of the output artefact to be published without using
    any wildcard syntax.  The output rule that provides this artefact should be
    specified using a must-match (`=`) prefix so that the job fails if it is
    not produced.  Each publish entry can specify exactly one output artefact.

* `series` **(string)**

Specify a series name to group a set of uploads together.  This is useful
    to group related files together in the URL space, even if they are produced
    by several different jobs.  This value should be short and URL-safe.

* `name` **(string)**

Specify the publically visible name of this file, which must be unique
    within the series for this commit for this repository.  This value should
    be short and URL-safe.

Each file published this way will be available at a predictable URL of the
  form:

```
  https://buildomat.eng.oxide.computer/public/file/OWNER/REPO/SERIES/VERSION/NAME
  ```

The `VERSION` value is the commit ID (full SHA) of the commit under test, and
  the `SERIES` and `NAME` come from the `publish` entry.

For example, if commit `e65aace9237833ec775253cfde97f59a0af5bc3d` from
  repository `oxidecomputer/software` included this publish directive:

```bash
  #: [[publish]]
  #: from_output = "/work/important-packaged-files.tar.gz"
  #: series = "packages"
  #: name = "files.tar.gz"
  ```

A published file would be available at the URL:

```
  https://buildomat.eng.oxide.computer/public/file/oxidecomputer/software/packages/e65aace9237833ec775253cfde97f59a0af5bc3d/files.tar.gz
  ```

Note that files published this way from private repositories will be
  available without authentication.

- `rust_toolchain` **(string or boolean)**

If specified, `rustup` will be installed in the environment and the nominated
  toolchain will be available as the default toolchain.  Any toolchain
  specification that `rustup` accepts should work here; e.g., something general
  like `stable` or `nightly`, or a specific nightly date, like
  `nightly-2022-04-27`.

```bash
  #: rust_toolchain = "stable"
  ```

It is also possible to use the boolean value `true` here, at which point
  the system will interpret the contents of the `rust-toolchain.toml` file
  in the root of the repository to decide what to install.  The file must
  contain a valid `channel` value, and may also contain a valid `profile`
  value.  Neither the legacy (pre-TOML) file format, nor TOML files which
  contain the `path` directive, are supported.

```bash
  #: rust_toolchain = true
  ```

- `skip_clone` **(boolean)**

By default, a basic job will clone the repository and check out the commit
  under test.  The working directory for the job will be named for
  the GitHub repository; e.g., for https://github.com/oxidecomputer/buildomat,
  the directory would be `/work/oxidecomputer/buildomat`.

If this option is specifed with the value `true`, no clone will be performed.
  The working directory for the job will be `/work` without subdirectories.
  This is useful in targets that do not provide toolchains or `git`, or where
  no source files from the repository (beyond the job program itself) are
  required for correct execution.

```bash
  #: skip_clone = true
  ```

- `target` **(string)**

The target for a job, which specifies the composition of the execution
  environment, can be specified by name.  Some targets (e.g., `lab`) are
  privileged, and not available to all repositories.

The list of unrestricted targets available for all jobs includes:

- `helios-latest`; an illumos execution environment (Oxide Helios
    distribution) running in an ephemeral virtual machine, with a reasonable
    set of build tools.  32GB of RAM and 200GB of disk should be available.
  - `omnios-r151038`; an illumos execution environment (OmniOS r151038 LTS)
    running in an ephemeral virtual machine, with a reasonable set of build
    tools.  32GB of RAM and 200GB of disk should be available.
  - `ubuntu-18.04`, `ubuntu-20.04`, and `ubuntu-22.04`; an Ubuntu execution
    environment running in an ephemeral virtual machine, with a reasonable set
    of build tools.  32GB of RAM and 200GB of disk should be available.

## Running Buildomat locally

The `cargo xtask local` set of commands helps you run a local buildomat.

The first thing to do is setting it up.  You will need [a configured AWS
profile][aws-profile] pointing to the AWS account that will host storage and
compute, and public DNS records pointing to ports 9979 (for `buildomat-server`)
and 4021 (for `buildomat-github-server`).  Once you have those, you can run the
setup:

```
$ cargo xtask local setup
```

The setup will ask which components to configure, ask you a few questions, and
create all the resources and configuration files you'll need.  Afterwards, you
can start the individual components:

* `cargo xtask local buildomat-server`
* `cargo xtask local buildomat-factory-aws`
* `cargo xtask local buildomat-github-server`

You can also interact with the local server using the CLI:

```
$ cargo xtask local buildomat COMMAND
```

[aws-profile]: https://docs.aws.amazon.com/cli/v1/userguide/cli-chap-authentication.html

## Licence

Unless otherwise noted, all components are licenced under the [Mozilla Public
License Version 2.0](./LICENSE).

buildomat

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README