KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#developer-tools #codex #typescript #ai-agents #claude-code #dsh #dsh-plugin #cli #deepseek-harness #macos #ai-agent #open-source

qos

★ 118 Open GitHub ↗

QuorumOS is a computation layer for running applications inside Trusted Execution Environments (TEEs)

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# QuorumOS

[![License: AGPL v3](https://img.shields.io/badge/License-AGPL%20v3-blue.svg)](https://github.com/tkhq/qos/blob/main/LICENSE) 
[![stagex-build Status](https://github.com/tkhq/qos/actions/workflows/stagex.yml/badge.svg)](https://github.com/tkhq/qos/actions/workflows/stagex.yml) 
[![stagex-build Status](https://github.com/tkhq/qos/actions/workflows/pr.yml/badge.svg)](https://github.com/tkhq/qos/actions/workflows/pr.yml)

## About

QuorumOS is a computation layer for running applications inside a Trusted Execution Environment ("TEE", also referred to as "enclave") at modern cloud scale. The OS architecture is based on the principle that a threshold of actors must coordinate to provision a secure compute environment with sensitive application logic and secret material; no single actor can unilaterally provision the environment or secret material.

Concretely, QuorumOS is designed to boot in an enclave by attesting to the enclave configuration, reconstructing a Quorum Key and then launching a single enclave app that can leverage the Quorum Key to encrypt and authenticate data.

The consensus on environment configuration is coordinated through the Manifest document which describes, among other things, the enclave image configuration, application CLI arguments, public Quorum Key, and quorum sets. During the bootstrapping process, a threshold of Quorum Members attest to the enclaves configuration against the Manifest out-of-band and then post their respective Quorum Key share. See the [provisioning](#provisioning) section for details.

The Quorum Key itself can be used by QuorumOS and enclave apps to encrypt and authenticate data.

QuorumOS ("QOS") is a minimal, immutable, and deterministic Linux unikernel targeting various Trusted Execution Environments for use cases that require high security and accountability.

For more information about how it's used within Turnkey, please refer to [The Turnkey Whitepaper](https://whitepaper.turnkey.com/), and more specifically: [Foundations](https://whitepaper.turnkey.com/foundations).

## Development

### Requirements

- 10GB+ free RAM
- Docker 26+
- GNU Make

### Reproducing builds

QuorumOS is built with [StageX](https://codeberg.org/stagex/stagex), a new deterministic Linux distro. StageX provides reproducible builds and guarantees a 1-to-1, immutable relationship between the human-readable source code in this repo and the resulting machine-executable artifacts produced by the build system.

This repository produces deterministic OCI container images. QuorumOS, the operating system, is packaged for execution inside of a Nitro EIF (Enclave Image File). This packaging is deterministic and done as part of [`qos_enclave`](./src/qos_enclave). The associated [Containerfile](./src/images/qos_enclave/Containerfile) contains the set of instructions to build the `nitro.eif` file, as well as `nitro.pcrs`, which contains the PCR measurements.

To produce the `qos_enclave` OCI container image, run:
```
make out/qos_enclave/index.json
```

If you need to extract files from it, you can do so by using [docker](https://docs.docker.com/get-started/get-docker/) and [skopeo](https://github.com/containers/skopeo):

```sh
# Creates an archive called qos_enclave.tar, with a tag "qos-enclave:latest"
skopeo copy oci:./out/qos_enclave:latest docker-archive:qos_enclave.tar:qos-enclave:latest

# Load the tar into local docker
docker load < qos_enclave.tar

# Create the container without running it (outputs a container ID)
docker create qos-enclave:latest

# Copy files locally for inspection
docker cp CONTAINER_ID:/nitro.pcrs nitro.pcrs

# Look at the PCR values
cat nitro.pcrs
b26733f9... PCR0
b26733f9... PCR1
21b9efbc... PCR2
```

These PCR values can be referenced against the content of [AWS remote attestations](https://docs.aws.amazon.com/enclaves/latest/user/set-up-attestation.html#pcr012).

### Submitting a PR

Before a PR can be merged our linter and unit tests need to pass.

Run them within [StageX](https://codeberg.org/stagex/stagex) (containerized) with:

```bash
make lint # apply standardized code formatting
make test # run unit tests locally
```

Run them locally (requires a local Rust installation) with:

```sh
make -C src lint
make -C src test
```

All tests must pass.

PRs also need to pass the `build-linux-only` job (part of the `pr` workflow). There are 3 crates excluded from the Rust workspace: `qos_system`, `qos_aws`, and `init`. These crates are excluded because they only build on Linux. If you are not working directly with these crates they generally only need to be updated if the dependencies for `qos_core` change. The linux only crates each have their own lockfile and that will need to be up to date for deterministic builds to work. To update the locks files you will need a linux build environment. Once in a linux build environment you can run `make build-linux-only`, which updates lock files if necessary; any updated lock files should then be committed.

### Running in qemu

NOTE: this has only been tested on a linux x86_64 host

#### Dependencies

Some additional dependencies are needed in order to run `qos` in the `nitro-enclave` machine in `qemu`.

1. [vhost-device-vsock](https://github.com/rust-vmm/vhost-device/blob/main/vhost-device-vsock/README.md) is required to provide the hypervisor link for the vsock
2. [qemu](https://www.qemu.org/) itself needs to be installed, `v11.0.0` or later

#### Egress support on host

To setup the egress tunnel you need root priviledges on the host machine and know your main internet route interface.
Run `src/scripts/enclave_egress_interfaces.sh <interface>` after each boot to provide the egress tunnel setup.

#### Running

Run `make qemu` to start a bare-bones qos instance that will await boot instructions.
This initiates local vsock on cid `1` and port `9001` for control and `9002` for egress and starts the enclave. The unix socket connector is at `/tmp/vhost4.socket`.

You can follow up by running `make host` and booting up the enclave.
You can also run the bridges by running `make bridge` which expects port `3000` to be used for the app host port.
Egress is enabled by default for the qemu setup and should start provided the manifest has egress defined.

To do a boot standard without needing manual steps, run `make boot`. You can override the pivot binary and arguments used for this by providing them in the `PIVOT_BIN` and `PIVOT_ARGS` env vars (don't forget that pivot args are in the format of `[arg1,arg2]`). The default program will attempt a download of a ~1MB file.

To stop the enclave and qemu run `make stop`.

## Releases

This project uses [`release-plz`](https://github.com/release-plz/release-plz). Install it with:

```sh
cargo install --locked release-plz
```

Once you have it installed you can try a release locally, to see what the release PR would be:

```
release-plz update
```

### Release flow

When PRs are merged to main, release-plz automatically opens a release PR if it doesn't exist yet, or updates the existing one.

The release PR is opened by the github-actions bot and tagged `release`. A **human** is expected to double check this PR and manually push any fixes to the release PR if necessary (usually: cosmetic changes to the CHANGELOGs).

Once the release PR is merged, the release-plz `release` workflow is triggered and all crates are published automatically.

**DO NOT** bump crates manually in standard PRs, let release-plz take care of this for you and edit the already-open release PR!

### Manually re-triggering a release

If something goes wrong during a release and you want to retry, you can open a PR tagged `release` and with a branch named `release-plz-****` to kick off the release workflow again (the `release` tag is necessary to trigger the workflow, and the branch name prefix is how release-plz determines that the PR is a release PR -- see [this](https://release-plz.dev/docs/config#the-release_always-field))

## Platforms

| Platform                   | Target  | Status   | Verified boot Method  |
|----------------------------|:-------:|:--------:|:---------------------:|
| Generic/Qemu               | generic | working  | Safeboot or Heads     |
| AWS Nitro Enclaves         | aws     | working  | Nitro attestation API |
| GCP Confidential Compute   | gcp     | research | vTPM 2.0 attestation  |
| Azure Confidential VMs     | azure   | research | vTPM 2.0 attestation  |

## Components

### Enclave

- houses server for listening to the Host
- contains logic for quorum key genesis, booting, and running an Enclave App
- see crate [`qos_core`](./src/qos_core/)

### Host

- EC2 instance housing the nitro enclave
- has a client for talking to nitro enclave
- has a server for incoming requests from outside world
- see crate [`qos_host`](./src/qos_host/)

#### Client

- anything making requests to host
- see crate [`qos_client`](./src/qos_client/)

## Key Terms

### Quorum Key

A set of 3 keys used for authentication, public key encryption, and encryption at rest. They are derived from a single seed. See the [QOS Key Set Specification](./src/qos_p256/SPEC.md) for details. The Quorum Key should only ever be reconstituted inside of an enclave. Additionally, the full provisioning of the Quorum Key concludes the attestation flow, at which point QuorumOS pivots to launching the specified enclave app. At rest, outside of the enclave, the key is intended to be stored across shares using shamir's secret sharing. Note that key shares are always intended to be stored encrypted to the Personal Key, never in plaintext.

You can learn more about how to use the quorum key in [Enclave Keys](docs/enclave_keys.md).

### Operator

An entity that may be a member of the [Manifest Set](#manifest-set) and/or [Share Set](#share-set). Operators use the P256 Signing and P256 HPKE schemes from the [QOS Key Set](./src/qos_p256/SPEC.md).

### Quorum Sets

There are two types of quorum sets:

### Manifest Set

The collection of members who can approve a manifest. In the typical instance provisioning flow, the Manifest Set will approve the details of the Manifest and then the Share Set members can trust a manifest if it has threshold approvals from the Manifest Set. This creates a logical division of responsibilities between Share Keys and Manifest Keys that enables better key usage practices and organizations to scale parts of the provision flow separately.

### Share Set

The collection of members who each hold shares of the Quorum Key and thus provision an enclave by attesting and then posting their shares. When posting shares, these members will also provide a signature of the manifest. The signature is recorded in the manifest envelope in order to leave an audit trail. This way, third parties can check which share set members actually participated in provisioning the quorum key

### Ephemeral Key

An asymmetric key generated by a QuorumOS instance. QOS uses a setup Ephemeral Key for provisioning and key forwarding, then rotates to a precommitted live Ephemeral Key for app-level use. Once Quorum Members verify setup attestation, they encrypt their Quorum Key shares to the setup Ephemeral Key and submit them to the instance for reconstruction. You can learn more about how to use ephemeral keys in [Enclave Keys](docs/enclave_keys.md).

### Manifest

A file that contains the static configuration to launch an instance of QuorumOS. The composition of the Manifest is attested to in the boot process. All Quorum Members will agree to the Manifest by signing it (QuorumOS should reject a submitted manifest if it has less than threshold signatures.)

### Node

A single machine compute instance running QuorumOS.

### Namespace

A group of QuorumOS Nodes running the same Enclave App and using the same Quorum Key. A Namespace contains many live Nodes all with the same Quorum Key and enclave app. Some of these nodes could be using different Manifests and different versions of the same enclave app.

### Pivot / Enclave App

The application QuorumOS pivots to once it finishes booting. This application's binary hash and CLI arguments are specified in the Manifest file.

## Provisioning

There are two modes for provisioning:

- [Boot Standard](docs/boot_standard.md): mode for provisioning an enclave with quorum key shares. This is required when there is no provisioned instance in the namespace to key forward from (first boot after genesis, or all nodes lost), when the manifest set or PCR3 (host IAM role) changes, or when a QuorumOS change breaks Key Forwarding itself. See [When Is Boot Standard Required?](docs/boot_standard.md#when-is-boot-standard-required) for the full decision rule.
- [Key Forward](docs/key_forward.md): mode for provisioning an enclave from an already provisioned enclave of the same namespace. This allows QuorumOS to support horizontal scaling cloud workloads. Effectively, a pre-existing enclave will verify attestation and manifest for a new enclave and then forward its quorum key to the new enclave.

### Remote Attestation

The purpose of remote attestation is to prove that an environment is running a particular piece of software. In the case of AWS Nitro Enclaves, an enclave can uniquely asks the Nitro Security Module (NSM) for an attestation document containing details of the enclave. This document is signed by the Nitro Attestation PKI and is tied back to the AWS Nitro Attestation PKI Root.
As defined in the [AWS documentation](https://docs.aws.amazon.com/enclaves/latest/user/verify-root.html) the instance can request the Nitro Security Module (NSM) to produce an attestation document on its behalf. QOS attestation requests carry the manifest hash in `user_data` and the relevant Ephemeral Key public key in `public_key`. Setup/provisioning attestations verify the setup key with PCR16; live/app attestations verify the live key with PCR17. Most App Proof verifiers only need the PCR17 live/app check, while PCR16 is for setup-time provisioning and key forwarding. Verifiers also check the AWS certificate chain, COSE signature, attestation freshness, expected nonce policy, and manifest PCR0 through PCR3.
Before provisioning a namespace with the Quorum Key, a Share Set will use the output of the attestation process against the enclave to verify that the enclave is running the expected version of QuorumOS and that that instance is configured in the expected manner as to warrant being provisioned with that Quorum Key.

Continued reading for attesting with nitro enclaves:

- <https://docs.aws.amazon.com/enclaves/latest/user/set-up-attestation.html>
- <https://docs.aws.amazon.com/enclaves/latest/user/verify-root.html>
- <https://docs.aws.amazon.com/enclaves/latest/user/nitro-enclave.html>

## Enclave Data Flow

See the [networking doc](docs/networking.md) for more info.

qos

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README