ci-mgmt

KO | EN

gitlite — search

#python #java #python3 #arduino #golang #machine-learning #rust #html #flask #javascript #seismology #nodejs

★ 17 Open GitHub ↗

Configuration for all things CI

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Pulumi CI Management

## Purpose

This repository contains code to manage CI/CD for the many Pulumi providers in a consistent and (mostly) automated manner. The repo's intended audience are Pulumi Corp engineers, but its contents may also serve as a helpful example for Pulumi community members looking to maintain their own providers with a similar CI/CD process to Pulumi Corp.

Pulumi providers use [GitHub Actions](https://docs.github.com/en/actions) for CI/CD. Because we maintain a long list of providers, we use this repository to:

- Generate GitHub Actions Workflow files for any provider. These can be deployed to all providers or a single provider respectively by the GitHub Actions workflows in this repository.
- Keep an [inventory of existing Pulumi providers](./provider-ci/providers).
- Maintain logic for branch protection across provider repositories.

## Usage

This repository has the following components:

- The `provider-ci` directory contains code to generate [GitHub Actions workflow files](https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions) for Pulumi providers, as well as the generated output for each provider (retained for the purpose of convenient output diffing).
- The `infra/providers/` directory contains a Pulumi program which uses the [Pulumi GitHub provider](https://www.pulumi.com/registry/packages/github/) to ensure consistent [branch protections](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/about-protected-branches) across our provider repositories.

For an overview of how Pulumi programs work, see [the Pulumi docs](https://www.pulumi.com/docs/).

- GitHub Actions workflows to automate common operations across all providers or a single provider.

## Prerequisites

The following tools are required for generating and deploying GitHub Actions workflows:

* [Mise](https://mise.jdx.dev/)

Dependencies required are modeled on `mise.toml`. Run `mise install` and `mise
settings experimental=true` (required for Go binaries such as `golangci-lint`
and `pulumictl`) to fetch and install them. Finally, run `mise env` to check if
env variables are being set correctly.

## Building

After checking out the code, run the following command:

```bash
cd provider-ci && make clean && make -j
```

Common commands:

- `make`: Generate all code and check the output.
- `make provider NAME=aws`: Generate code for single provider with debug information
- `make lint-providers`: Check the generated code for all providers.
- `make lint-providers/aws/repo`: Check the generated code for a specific provider.

### Adding generated workflows to your local provider repository

Sometimes, you want those changes NOW rather than having to wait for a GitHub PR.
This example command will generate workflows for pulumi-datadog, and place them in the specified `--out` directory.
Adjust for your provider and filesystem.

```bash
./bin/provider-ci generate --name pulumi/pulumi-datadog --template bridged-provider --config ./providers/datadog/config.yaml --out ../../pulumi-dtadog
```

## Adding a New Bridged Provider

To add a new provider:

1. A new provider needs a top-level `.ci-mgmt.yaml` in its _own_ repository with the following basic configuration:

```yaml
   # Required values:
   provider: foo # substitute the name of your provider, without the pulumi- prefix
   env: # A map of required configuration for any integration tests, etc.
     AN_OPTION: value
     ANOTHER_OPTION: true
     # etc.
   lint: true # Linting should be true in most cases, unless failing rules in the upstream provider makes this impractical.

# Optional values:
   docker: true # Whether the provider's tests use Docker to run. If set to true, a file `testing/docker-compose.yml` must be present in the provider repository.
   setup-script: testing/setup.sh # Path to a script that's used for testing bootstraps
   ```

`ci-mgmt` will read your provider's `.ci-mgmt.yaml` and generate the standard set of CI files from templates.
   You can override every one of the [default values](./provider-ci/internal/pkg/templates/bridged-provider.config.yaml)
   in your `.ci-mgmt.yaml` file.

1. Add your provider to `provider-ci/providers.json` in alphabetical order. This ensures your provider receives regular
   updates and maintenance.

1. Add your provider to `.github/ISSUE_TEMPLATE/0-ecosystem-providers.md` to ensure manual rollouts track your provider.

1. Commit the changes and open a pull request.

1. To receive a pull request with your new config files, you can run the
   [Update Workflows, Single Bridged Provider](https://github.com/pulumi/ci-mgmt/actions/workflows/update-workflows-single-bridged-provider.yml)
   workflow run, using your provider name as the input.
   Another option is to wait for the nightly cronjob to send this pull request automatically.

1. If you would like to manually generate the configuration to get started right away, you can do so in your provider
   repository root:

```bash
   go run github.com/pulumi/ci-mgmt/provider-ci@master generate \
      --name pulumi/pulumi-$(PROVIDER_NAME) \
      --out . \
      --template bridged-provider \
      --config .ci-mgmt.yaml
   ```

The generated files will be writen to your current directory.

## Customizing SDK Generation (`sdk-hooks.mk`)

Providers occasionally need to fine-tune `pulumi package gen-sdk` (or legacy codegen) output, for example to delete a stray generated file or patch a type stub after each regeneration. Because ci-mgmt regenerates and overwrites the `Makefile`, editing the generated recipes directly is not regen-safe. The generated `Makefile` therefore exposes a provider-owned escape hatch: an optional `sdk-hooks.mk`.

### Contract

The generated `Makefile` ends with `-include sdk-hooks.mk` (the leading `-` means no error if the file is absent), and invokes two slots around each per-language SDK codegen recipe, in **both** the `gen-sdk` and the legacy codegen branches:

- `PRE_GEN_SDK_<LANG>` runs immediately before the codegen step.
- `POST_GEN_SDK_<LANG>` runs immediately after codegen, before the recipe's completion sentinel is touched.

`<LANG>` is the uppercased SDK language, giving ten slots in total: `PRE_GEN_SDK_GO` / `POST_GEN_SDK_GO`, and likewise for `NODEJS`, `PYTHON`, `DOTNET`, and `JAVA`. Each slot is a plain make variable expanded as a recipe line, so an unset slot expands to nothing and make skips it entirely. With no `sdk-hooks.mk` present, every slot is unset and behavior is identical to today.

A slot's value is a shell command run from the provider repo root (the same working directory as the codegen recipe). For anything beyond a single command, point the slot at a checked-in script so the logic stays readable and testable.

### Opt-in and regen-safe

`ci-mgmt` never creates `sdk-hooks.mk`; it is entirely opt-in, and regenerating CI does not touch it. To adopt it, copy [`provider-ci/sdk-hooks.mk.example`](./provider-ci/sdk-hooks.mk.example) into your provider repo root as `sdk-hooks.mk`, keep only the slots you need, and commit it.

### Worked example

`pulumi-pulumiservice` needs to drop a generated file from the Go SDK and patch a Python stub after each regeneration. With this `sdk-hooks.mk` in the repo root:

```makefile
POST_GEN_SDK_GO     = rm -f sdk/go/pulumiservice/internal/unused.go
POST_GEN_SDK_PYTHON = ./scripts/patch-python-stubs.sh
```

running `make generate_go` deletes the stray file right after gen-sdk emits it, and `make generate_python` runs the stub patcher. No other language is affected, and removing `sdk-hooks.mk` restores stock behavior.

## OpenInspect Setup

Internal provider templates generate the `.openinspect` directory so OpenInspect sandboxes have a consistent bootstrap path. The shared `internal` template owns it.

The common setup installs and trusts `mise`, then runs optional setup hooks from `.openinspect/setup.d/*.py` and `.openinspect/setup.local.py`. `start.py` runs the equivalent hooks from `.openinspect/start.d/*.py` and `.openinspect/start.local.py`. More specific templates can add generated hooks under either directory; for example, `internal-bridged` adds a setup hook that runs `make prepare_local_workspace`.

Setup and start answer different questions, and neither implies the other. Setup runs when an image is built, and on a fresh boot with no image — it is what gets baked. Start runs on every session boot except image builds — it is what has to be true for the session in front of you. A fresh session with no image is the only case where both run.

Their time budgets differ accordingly — setup gets a long one, start a short one — so expensive work belongs in `setup.d` and `start.d` should stay a quick session entrypoint. See the generated `.openinspect/README.md` for the hook-authoring details, including the stdout trap that makes a backgrounded process look like a hung start hook.

Provider repositories should not edit generated `.openinspect` files directly. To customize generated settings, add values to `.ci-mgmt.yaml`:

```yaml
openinspect:
  settings:
    exampleField: exampleValue
```

To customize setup or start behavior, add provider-owned hook files such as `.openinspect/setup.local.py`, `.openinspect/setup.d/90-provider-setup.py`, or their `start` equivalents. A failed start hook is fatal to the session, so a hook whose failure the session can survive should catch its own errors.

## Updating All Bridged Providers

The [Update GH Workflows, ecosystem providers](https://github.com/pulumi/ci-mgmt/actions/workflows/update-workflows-ecosystem-providers.yml)
Workflow runs on a nightly schedule.
You may trigger this Workflow manually; however be aware that this causes a lot of GitHub Actions to run at the same
time, which may cause rate limiting across the org. Plan ahead and do this at a low-traffic time.

## Updating GitHub workflow schema

Fetch the latest JSON Schema then re-generate type definitions:

```bash
make discovery
```

## Automatically editing source code across provider repositories

You can apply ad-hoc source edits across provider repositories even on files that are not managed or generated by ci-mgmt. For example, you might need to automatically update example code to use a newer Pulumi SDK major version dependency or a newer version of the underlying infrastructure such as .NET Framework.  This can be done with migrations:

- describe your desired edits as a `SourceMigration` in `provider-ci/internal/pkg/migrations`

- test your changes locally by running ci-mgmt:

```bash
   go run . generate -c ../../pulumi-azure/.ci-mgmt.yaml -o ../../pulumi-azure/
   ```

- stand up a PR to ci-mgmt

- trigger an action such as
  [update-workflows-bridged-providers.yml](https://github.com/pulumi/ci-mgmt/actions/workflows/update-workflows-bridged-providers.yml)
  from the PR; this will create PRs that synchronize the selected repositories with ci-mgmt and apply the source
  migration as part of the change

- merge the PR to ci-mgmt

- merge the PR to the desired provider repositories (not needed if these PRs are set to automerge)

ci-mgmt

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README