KO
|
EN
gitlite — search
Search
#javascript
#typescript
#api
#python
#php
#game
#generator
#grpc
#docker-compose
#flutter
#chrome-extension
#nodejs
buildomat
★ 69
Open GitHub ↗
a software build labour-saving device
Download README (.md)
Explore Similar Repositories
ImguiCandy
:
Color utils, Themes and other cool stuff for Dear ImGui
HandPosing_Demo
:
A Pirate demo for the HandPosing tool
Niconicome
:
ニコニコ動画のコンテンツをダウンロードします
truly_shift_invariant_cnns
:
No description available.
IAmLazy
:
Easily backup and restore your tweaks
// repository documentation
Was this content helpful?
★ 0
(0 ratings)
Select Rating:
★
★
★
★
★
Submit Feedback
Recent Feedback
×
Download README
Do you want to download the
README.md
file for
buildomat
?
Download (.md)
<p align="center"> <img src="./www/buildomat_wide.png" alt="hammer logo"><br> <b>B U I L D O M A T</b><br> a software build labour-saving device<br> </p> <hr> Buildomat manages the provisioning of ephemeral UNIX systems (e.g., instances in AWS EC2) on which to run software builds. It logs job output, collects build artefacts, and reports status. The system integrates with GitHub through the Checks API, to allow build jobs to be triggered by pushes and pull requests. ## Components Buildomat is made up of a variety of crates, loosely grouped into areas of related functionality: ``` $ cargo xtask crates buildomat /bin buildomat-agent /agent buildomat-bunyan /bunyan buildomat-client /client buildomat-common /common buildomat-database /database buildomat-server /server buildomat-types /types buildomat-factory-aws /factory/aws buildomat-factory-lab /factory/lab buildomat-github-common /github/common buildomat-github-database /github/database buildomat-github-dbtool /github/dbtool buildomat-github-ghtool /github/ghtool buildomat-github-server /github/server xtask /xtask ``` ### Buildomat Core The buildomat core is responsible for authenticating users and remote services, for managing build systems, and for running jobs and collecting output. #### Server (`buildomat-server`, in `server/`) The core buildomat API server. Coordinates the creation, tracking, and destruction of workers in which to execute jobs. This component sits at the centre of the system and is used by the GitHub integration server, the client command, the agent running within each worker for control of the job, and any factories. #### Client Command (`buildomat`, in `bin/`) A client tool that uses the client library to interface with and manipulate the core server. The tool has both administrative and user-level functions, expressed in a relatively regular hierarchy of commands; e.g., `buildomat job run` or `buildomat user ls`. ``` $ ./target/release/buildomat Usage: buildomat [OPTS] COMMAND [ARGS...] Commands: info get information about server and user account control server control functions job job management user user management Options: --help usage information -p, --profile PROFILE authentication and server profile ERROR: choose a command ``` #### Client Library (`buildomat-client`, in `client/`) A HTTP client library for accessing the core buildomat server. This client is generated at build time by [`progenitor`](https://github.com/oxidecomputer/progenitor), an OpenAPI client generator. The client is generated based an OpenAPI document managed in the repository and generated by Dropshot based on the implementation of the server and then checked in to the repository. If you make changes to the API exposed by the core server, you will need to regenerate the document, `client/openapi.json`, using: ``` $ cargo xtask openapi ``` #### Agent (`buildomat-agent`, in `agent`/) A process that is injected into an ephemeral AWS EC2 instance to allow the buildomat core server to take control and run jobs. This process receives single-use credentials at provisioning time from the core server, and connects out to receive instructions. The agent does not require a public IP, or any direct inbound connectivity, to allow agents to run inside remote NAT environments. ### Factories Buildomat jobs are specified to execute within a particular target environment. Concrete instances of those target environments (known as _workers_) are created, managed, and destroyed by _factories_. Factories are long-lived server processes that connect to the core API and offer to construct workers as needed. When a worker has finished executing the job, or when requested by an operator, the factory is also responsible for freeing any resources that were in use by the worker. #### AWS Factory (`buildomat-factory-aws` in `factory/aws/`) The AWS factory creates ephemeral AWS instances that are used to run one job and are then destroyed. The factory arranges for the agent to be installed and start automatically in each instance that is created. The factory is responsible for ensuring no stale resources are left behind, and for enforcing a cap on the concurrent use of resources at AWS. Each target provided by an AWS factory can support a different instance type (i.e., CPU and RAM capacity), a different image (AMI), and a different root disk size. #### Lab Factory (`buildomat-factory-lab` in `factory/lab/`) The lab factory uses IPMI to exert control over a set of physical lab systems. When a worker is required, a lab system is booted from a ramdisk and the agent is started, just as it would be for an AWS instance. From that point on, operation is quite similar to AWS instances: the agent communicates directly with the core API. When tearing down a lab worker, the machine is rebooted (again via IPMI) to clear out the prior ramdisk state. Each target provided by a lab factory can boot from a different ramdisk image stored on a local server. ### GitHub Integration (formerly known as Wollongong) The GitHub-specific portion of the buildomat suite sits in front of the core buildomat service. It is responsible for receiving and processing notifications of new commits and pull requests on GitHub, starting any configured build jobs, and reporting the results so that they are visible through the GitHub user interface. #### Server (`buildomat-github-server`, in `github/server/`) This server acts as a [GitHub App](https://docs.github.com/en/developers/apps). It is responsible for processing incoming [GitHub webhooks](https://docs.github.com/en/developers/webhooks-and-events/webhooks/about-webhooks) that notify the system about commits and pull requests in authorised repositories. In addition to relaying jobs between GitHub and the buildomat core, this service provides an additional HTML presentation of job state (e.g., detailed logs) and access to any artefacts that jobs produce. This server keeps state required to manage the interaction with GitHub, but does not store job data; requests for logs or artefacts are proxied back to the core server. #### Database Tool (`buildomat-github-dbtool`, in `github/dbtool/`) This tool can be used to inspect the database state kept by the GitHub integration as it tracks GitHub pull requests and commits. Unlike the core client tool, this program directly interacts with a local SQLite database. ``` $ buildomat-github-dbtool Usage: buildomat-github-dbtool COMMAND [ARGS...] Commands: delivery (del) webhook deliveries repository (repo) GitHub repositories check GitHub checks Options: --help usage information ERROR: choose a command ``` Of particular note, the tool is useful for inspecting and replaying received webhook events; e.g., ``` $ buildomat-github-dbtool del ls SEQ ACK RECVTIME EVENT ACTION 0 1 2021-10-05T01:58:32Z ping - 1 1 2021-10-05T02:25:33Z installation created 2 1 2021-10-05T02:26:53Z push 3 1 2021-10-05T02:26:53Z check_suite requested 4 1 2021-10-05T02:26:56Z check_suite completed 5 1 2021-10-05T02:26:56Z check_run completed 6 1 2021-10-05T02:26:56Z check_run created 7 1 2021-10-05T02:26:56Z check_run created 8 1 2021-10-05T02:26:57Z check_run created ... ``` The `buildomat-github-dbtool del unack SEQ` command can be used to trigger the reprocessing of an invididual webhook message. ## Per-repository Configuration Buildomat works as a [GitHub App](https://docs.github.com/en/developers/apps), which is generally "installed" at the level of an [Organisation](https://docs.github.com/en/organizations). Installing the App allows buildomat to receive notifications about events, such as git pushes and pull requests, from all repositories (public and private) within the organisation. In order to avoid accidents, buildomat requires that the service be explicitly configured for a repository before it will take any actions. Per-repository configuration is achieved by creating a file **in the default branch** (e.g., `main`) of the repository in question, named `.github/buildomat/config.toml`. This file is written in [TOML](https://toml.io/), with a handful of simple values. Supported properties in this file include: - `enable` **(boolean)** Must be present and have the value `true` in order for buildomat to consider the repository for jobs; e.g., ```toml enable = true ``` - `org_only` **(boolean, defaults to `true` if missing)** If set to `true`, or missing from the file, buildomat will not automatically run jobs in response to pull requests opened by users that are not a member of the GitHub Organisation which owns the repository. If set to `false`, any GitHub user can cause a job to be executed. This property is important for security if your repository is able to create any jobs that have access to secrets, or to restricted networks. - `allow_users` **(array of strings, each a GitHub login name)** If specified, jobs will be started automatically for users in this list, regardless of whether they are a member of the Organisation that owns the repository or not, and regardless of the value of the `org_only` property. This is often useful for pre-authorising jobs driven by Pull Requests made by various automated systems; e.g., ```toml allow_users = [ "dependabot[bot]", "renovate[bot]", ] ``` Note that buildomat will only ever read this configuration file from the most recent commit in the default branch of the repository, not from the contents of another branch or pull request. This is of particular importance for security-sensitive properties like `org_only`, where the policy set by users with full write access to the repository must not be overridden by changes from potentially untrusted users. If a pull request with a malicious policy change is merged, it will then be in the default branch and active for subsequent pull requests; maintainers must carefully review pull requests that change this file. ## Specifying Jobs Once you have configured buildomat at the repository level, you can specify some number of jobs to execute automatically in response to pushes and pull requests. While per-repository configuration is read from the default branch, jobs are read from the commit under test. Jobs are specified as `bash` programs with some configuration directives embedded in comments. These job files must be named `.github/buildomat/jobs/*.sh`. Unexpected additional files in `.github/buildomat/jobs` will result in an error. Job files should begin with an interpreter line, followed by TOML-formatted configuration prefixed with `#:` so that they will be identified as configuration by buildomat, but ignored by the shell. For example, a minimal job that would just execute `uname -a`: ```bash #!/bin/bash #: #: name = "build" #: variety = "basic" #: uname -a ``` The minimum set of properties that must always appear in the TOML frontmatter is: - `name` **(string)** Must be present in all jobs. This name is used for at least two things: as the name of the Check Run in the GitHub user interface, and when specifying that some other job depends on this job. The job name must be unique amongst all jobs within the commit under test. In general, it is probably best to keep these short, lower-case, and without spaces. It is conventional to use the same name for the job file and the job, e.g., `name = "build"` in file `.github/buildomat/jobs/build.sh`. - `variety` **(string)** To allow the system to evolve over time, a job must specify a _variety_, which defines some things about the way a job executes and what additional configuration options are required or available. These properties are optional, but not variety-specific: - `enable` **(boolean)** To exclude a particular job file from processing, set this to `false`. If not specified, this property defaults to `true`. This allows a job to be temporarily disabled without needing to be removed from the repository. The rest of the configuration is variety-specific. ### Variety: Basic Each **basic** variety job (selected by specifying `variety = "basic"` in the frontmatter) takes a single `bash` program and runs it in an ephemeral environment. The composition of that environment, such as compute and memory capacity or the availability of specific toolchains and other software, depends on the `target` option. Basic variety jobs can produce output files (see the configuration options `output_rules` and `publish`). They can also depend on the successful completion of other jobs, gaining access to any output files from the upstream job (see the `dependencies` option). Jobs are generally executed in parallel, unless they are waiting for a dependency or for capacity to become available. #### Execution Environment By default, an ephemeral system (generally a virtual machine) will be provisioned for each job. The system will be discarded at the end of the job, so no detritus is left behind. Once the environment is provisioned, the `bash` program in the job file is executed as-is. Jobs are executed as an unprivileged user, `build`, with home directory `/home/build`. If required, this user is able to escalate to `root` privileges through the use of [pfexec(1)](https://illumos.org/man/1/pfexec). Systems that do not have a native `pfexec` will be furnished with a compatible wrapper around a native escalation facility, to ease the construction of cross-platform jobs. By default, the working directory for the job is based on the name of the repository; e.g., for https://github.com/oxidecomputer/buildomat, the working directory would be `/work/oxidecomputer/buildomat`. The system will arrange for the repository to be cloned at that location with the commit under test checked out. A simple job could directly invoke some build tool like `gmake` or `cargo build`, and the build would occur at the root of the clone. The `skip_clone` configuration option can disable this behaviour. Most targets provide toolchains from common metapackages like `build-essential`; e.g., `gmake` and `gcc`. If a Rust toolchain is required, one can be requested through the `rust_toolchain` configuration option. This will be installed using [rustup](https://rustup.rs/). ##### Environment Variables While the complete set of environment variables is generally target-specific, the common minimum for all targets includes: - `BUILDOMAT_JOB_ID` will be set to the unique ID of this job - `CI` will be set to `true` - `GITHUB_REPOSITORY` set to `owner/repository`; e.g., `oxidecomputer/buildomat` - `GITHUB_SHA` set to the commit ID of the commit under test - If the commit under test is part of a branch, then `GITHUB_BRANCH` will be set to the branch name (e.g., `main`) and `GITHUB_REF` will be set to the ref name; e.g., `res/heads/main`. - `HOME`, set to the home directory of the build user - `USER` and `LOGNAME`, set to the username of the build user - `PATH` set to include relevant directories for toolchains and other useful software - `TZ` will be set to `UTC` - `LANG` and `LC_ALL` will be set to `en_US.UTF-8` ##### Available Commands Cross-platform shell programming can be challenging due to differences between different operating systems. To make this a little easier, we ensure that each buildomat target can provide a basic suite of tools that are helpful in constructing succint jobs: - [pfexec(1)](https://illumos.org/man/1/pfexec) allows escalation from the unprivileged build user to `root`; e.g., `pfexec id -a`. - [ptime(1)](https://illumos.org/man/1/ptime) runs a program and provides (with `-m`, detailed) timing information; e.g., `ptime -m cargo test`. - [banner(1)](https://illumos.org/man/1/banner) prints its arguments in large letters on standard output, and is useful for producing headings in job log output; e.g., `banner testing`. #### Configuration Configuration properties supported for basic jobs include: - `access_repos` **(array of strings)** Jobs can be created in both public and private repositories. Public repositories are available to everybody, but private repositories require appropriate credentials. By default, an ephemeral, read-only token is injected into the execution environment (in the `$HOME/.netrc` file) that is able to access only the repository directly under test. If a job requires access to additional private repositories beyond the direct repository, they may be specified in this list, in the form `owner/repository`; e.g., ```bash #: access_repos = [ #: "oxidecomputer/clandestine-project", #: "oxidecomputer/secret-plans", #: ] ``` Note that this option only works for repositories within the same organisation as the direct repository. Using the option will trigger a requirement for job-level authorisation by a member of the organisation. - `dependencies` **(table)** A job may depend on the successful completion of one or more other jobs from the same commit under test. If the dependency is cancelled or fails to complete successfully for some other reason, that failure will be propagated forward as a failure of this job. Each entry in the dependencies table is itself a table with a name for the dependency, and the following per-dependency properties: * `job` **(string)** Specifies the job that this job should wait on for execution. The `job` value must exactly match the `name` property of some other `basic` variety job available in the same commit. Any artefacts output by the job named in the dependency will be made available automatically under `/input/$dependency` using the dependency name. For example, consider this dependency directive: ```bash #: [dependencies.otherjob] #: job = "the-other-job!" ``` If the job with the name `the-other-job!` produces an output file, `/tmp/output.zip`, then it will be made available within this job as the file `/input/otherjob/tmp/output.zip`. Using this facility, one can easily split a job into a "build" phase that runs under a target with access to toolchains, and one or more "test" phases that can take the build output and run it in under another target that might not have a toolchain or may have access to other resources that have limited availability like test hardware. Jobs can also depend on more than one other job, allowing a job to aggregate artefacts from several other jobs together in one place. This might be useful when building binaries for more than one different OS, with a final step that publishes multi-OS packages if all the other builds were successful. Cycles in the dependency graph are not allowed. - `output_rules` **(array of strings)** Jobs may produce artefacts that we wish to survive beyond the lifetime of the ephemeral build environment. A job may specify one or more files for preservation by the system; e.g., a build job may produce binaries or packages that can then be downloaded and installed, or a test job may produce JUnit XML files or other diagnostic logs that can be inspected by engineers. The `output_rules` property is a list of `/`-anchored glob patterns that match files in the ephemeral machine; e.g., `/tmp/*.txt` would match `/tmp/something.txt` but not `/tmp/nothing.png`. Like the shell, a single asterisk (`*`) will not descend into a hierarchy of directories. If you want to match recursively, a double asterisk (`**`) pattern will match the current directory or any directory under that directory, but not files. You can combine these to get a recursive match; e.g., `/tmp/**/*.txt` would match `/tmp/a.txt`, `/tmp/dir/a.txt`, and `/tmp/dir/dir/a.txt`. By default, it is not an error to specify a pattern that does not match any files. Provided the job is not cancelled, matching files are uploaded whether the job program exits with a zero status (denoting success) or a non-zero status (denoting failure). These behaviours can be used to upload diagnostic logs left behind by unexpected test failures that are cleaned up on success; e.g., ```bash #: output_rules = [ #: "/tmp/test_output/*", #: ] ``` If the success of a job _requires_ that a particular artefact is produced, the `=` prefix may be used to signify "this rule must match at least one file". If the rule does not match at least one output file, the job is marked as failed even if the job program otherwise succeeded. This can be used to make sure that, say, a release binary build job produces an archive with the expected name; e.g., ```bash #: output_rules = [ #: "=/work/pkg/important.tar.gz", #: "=/work/pkg/important.sha256.txt", #: ] ``` By default, the system attempts to ensure that a job has not accidentally left background processes running that continue to modify the output artefacts. If the size or modified time of a file changes while it is being uploaded, the job will fail. To relax this restriction, the `%` prefix may be used to signify that "this file is allowed to change while it is being uploaded". The `%` prefix will also ignore a file that is completely removed by a background process before it is able to be uploaded. This is used to make best effort uploads of diagnostic log files for background processes which may continue running even though the job is nominally complete; e.g., ```bash #: output_rules = [ #: "%/var/svc/log/*.log", #: ] ``` To exclude specific files from upload, the `!` prefix can be used to signify that "any file that matches this pattern should be ignored, even if it was nominally included by another pattern". Order in the array is not important; a match of any exclusion rule will prevent that file from behing uploaded. For example, to upload anything left in `/tmp` except for pictures: ```bash #: output_rules = [ #: "/tmp/*", #: "!/tmp/*.jpg", #: ] ``` The must-match (`=`) and allow-change (`%`) prefixes may be combined in a single output rule. The exclusion prefix (`!`) may not be combined with any other prefix. For example, to require at least one log file (which may still be growing) that is not `big-and-useless.log`: ```bash #: output_rules = [ #: "=%/tmp/*.log", #: "!/tmp/big-and-useless.log", #: ] ``` - `publish` **(array of tables)** Some jobs may wish to publish a specific subset of their output artefacts at a predictable URL based on the commit ID of the commit under test, for reference by other jobs from other repositories, or end user tools. Each table in the `publish` array of tables must contain these properties: * `from_output` **(string)** Specify the full path of the output artefact to be published without using any wildcard syntax. The output rule that provides this artefact should be specified using a must-match (`=`) prefix so that the job fails if it is not produced. Each publish entry can specify exactly one output artefact. * `series` **(string)** Specify a series name to group a set of uploads together. This is useful to group related files together in the URL space, even if they are produced by several different jobs. This value should be short and URL-safe. * `name` **(string)** Specify the publically visible name of this file, which must be unique within the series for this commit for this repository. This value should be short and URL-safe. Each file published this way will be available at a predictable URL of the form: ``` https://buildomat.eng.oxide.computer/public/file/OWNER/REPO/SERIES/VERSION/NAME ``` The `VERSION` value is the commit ID (full SHA) of the commit under test, and the `SERIES` and `NAME` come from the `publish` entry. For example, if commit `e65aace9237833ec775253cfde97f59a0af5bc3d` from repository `oxidecomputer/software` included this publish directive: ```bash #: [[publish]] #: from_output = "/work/important-packaged-files.tar.gz" #: series = "packages" #: name = "files.tar.gz" ``` A published file would be available at the URL: ``` https://buildomat.eng.oxide.computer/public/file/oxidecomputer/software/packages/e65aace9237833ec775253cfde97f59a0af5bc3d/files.tar.gz ``` Note that files published this way from private repositories will be available without authentication. - `rust_toolchain` **(string or boolean)** If specified, `rustup` will be installed in the environment and the nominated toolchain will be available as the default toolchain. Any toolchain specification that `rustup` accepts should work here; e.g., something general like `stable` or `nightly`, or a specific nightly date, like `nightly-2022-04-27`. ```bash #: rust_toolchain = "stable" ``` It is also possible to use the boolean value `true` here, at which point the system will interpret the contents of the `rust-toolchain.toml` file in the root of the repository to decide what to install. The file must contain a valid `channel` value, and may also contain a valid `profile` value. Neither the legacy (pre-TOML) file format, nor TOML files which contain the `path` directive, are supported. ```bash #: rust_toolchain = true ``` - `skip_clone` **(boolean)** By default, a basic job will clone the repository and check out the commit under test. The working directory for the job will be named for the GitHub repository; e.g., for https://github.com/oxidecomputer/buildomat, the directory would be `/work/oxidecomputer/buildomat`. If this option is specifed with the value `true`, no clone will be performed. The working directory for the job will be `/work` without subdirectories. This is useful in targets that do not provide toolchains or `git`, or where no source files from the repository (beyond the job program itself) are required for correct execution. ```bash #: skip_clone = true ``` - `target` **(string)** The target for a job, which specifies the composition of the execution environment, can be specified by name. Some targets (e.g., `lab`) are privileged, and not available to all repositories. The list of unrestricted targets available for all jobs includes: - `helios-latest`; an illumos execution environment (Oxide Helios distribution) running in an ephemeral virtual machine, with a reasonable set of build tools. 32GB of RAM and 200GB of disk should be available. - `omnios-r151038`; an illumos execution environment (OmniOS r151038 LTS) running in an ephemeral virtual machine, with a reasonable set of build tools. 32GB of RAM and 200GB of disk should be available. - `ubuntu-18.04`, `ubuntu-20.04`, and `ubuntu-22.04`; an Ubuntu execution environment running in an ephemeral virtual machine, with a reasonable set of build tools. 32GB of RAM and 200GB of disk should be available. ## Running Buildomat locally The `cargo xtask local` set of commands helps you run a local buildomat. The first thing to do is setting it up. You will need [a configured AWS profile][aws-profile] pointing to the AWS account that will host storage and compute, and public DNS records pointing to ports 9979 (for `buildomat-server`) and 4021 (for `buildomat-github-server`). Once you have those, you can run the setup: ``` $ cargo xtask local setup ``` The setup will ask which components to configure, ask you a few questions, and create all the resources and configuration files you'll need. Afterwards, you can start the individual components: * `cargo xtask local buildomat-server` * `cargo xtask local buildomat-factory-aws` * `cargo xtask local buildomat-github-server` You can also interact with the local server using the CLI: ``` $ cargo xtask local buildomat COMMAND ``` [aws-profile]: https://docs.aws.amazon.com/cli/v1/userguide/cli-chap-authentication.html ## Licence Unless otherwise noted, all components are licenced under the [Mozilla Public License Version 2.0](./LICENSE).
