KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#typescript #ai-agents #cli #claude-code #codex #developer-tools #dsh-plugin #deepseek-harness #windows #open-source #agent-skills #ai

buildpacks-ci

★ 41 Open GitHub ↗

Concourse CI pipelines for the buildpacks team

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# Introduction

This contains the configuration for the Cloud Foundry Buildpacks team [Concourse deployment](https://buildpacks.ci.cf-app.com/).

# Pipelines

All pipelines are now generated using [ytt](https://carvel.dev/ytt/) templating.

* [dependency-builds](pipelines/dependency-builds): build binaries for Cloud Foundry buildpacks
* [buildpacks](pipelines/buildpack): test and release all of the buildpacks
* [brats](pipelines/brats): run [BRATS](https://github.com/cloudfoundry/brats) against the master branch of buildpacks
* [buildpack-verification](pipelines/buildpack-verification): generate static site for buildpack verification
* [cflinuxfs4](pipelines/cflinuxfs4.yml): cflinuxfs4 rootfs pipeline

# Concourse State

Jobs and tasks in the `buildpacks-ci` repository store state in [public-buildpacks-ci-robots](https://github.com/cloudfoundry/public-buildpacks-ci-robots). See repository README for details.

# Commands and recipes

## Updating Pipelines

After making changes to pipeline templates (ytt files), regenerate and deploy them to Concourse.

### Update All Buildpack Pipelines
```sh
./bin/update-pipelines --buildpacks-only
```
Updates: python-buildpack, ruby-buildpack, go-buildpack, nodejs-buildpack, php-buildpack, java-buildpack, and all other buildpacks.

### Update a Specific Pipeline
```sh
./bin/update-pipelines -p python-buildpack
```

### Update All Pipelines
```sh
./bin/update-pipelines --all
```
Updates buildpack pipelines and infrastructure pipelines (dependency-builds, cf-release, cflinuxfs4, etc.)

### Validate Without Deploying (Dry Run)
```sh
./bin/update-pipelines -p python-buildpack --dry-run
```

### List Available Pipelines
```sh
./bin/update-pipelines --list
```

### When to Update Pipelines

✅ **Required after these changes:**
- Merging changes to pipeline templates (`pipelines/buildpack/pipeline.yml`)
- Merging changes to ops files or pipeline structure
- After cf-deployment version changes affecting pipeline configuration

❌ **NOT required for:**
- Changes to task files (`tasks/*/task.yml`) - these are fetched from git during job execution
- Changes to scripts or other non-pipeline files

## Debugging the build

```sh
fly intercept -j $JOB_NAME -t task -n $TASK_NAME
```

## Clearing the git resources

```sh
fly intercept -c $RESOURCE_NAME rm -rf /tmp/git-resource-repo-cache
```

## To build a new version of a binary

1. Check out the `binary-builds` branch
2. Edit the YAML file appropriate for the build (e.g. `ruby-builds.yml`)
3. Find the version number and package SHA256 of the new binary. For many binaries, the project website provides the SHA256 along with the release (for example, jruby.org/download provides the SHA256 along with each JRuby release). For others (such as Godep), you download the .tar.gz file and run `shasum -a 256 <tar_file>` to obtain the SHA256.
4. Add any number of versions and their checksums to the array, e.g.

```yaml
	ruby:
	- version: 2.2.2
	  sha256: 5ffc0f317e429e6b29d4a98ac521c3ce65481bfd22a8cf845fa02a7b113d9b44
	```

5. `git commit -am 'Build ruby 2.2.2' && git push`

Build should automatically kick off at
https://buildpacks.ci.cf-app.com/pipelines/binary-builder and silently
upload a binary to the `buildpacks.cloudfoundry.org` bucket under
`dependencies/`,
e.g. https://buildpacks.cloudfoundry.org.s3.amazonaws.com/dependencies/ruby/ruby-2.2.2-linux-x64.tgz

Note that the array is a stack, which will be emptied as the build
succeeds in packaging successive versions.

## Making Changes to Build Scripts

When you want to change how a binary gets built, there are two places you may need to make changes. All binaries are built by the `dependency-builds` pipeline, and you may need to change the task that builds them. For many binaries, the `dependency-builds` pipeline runs recipes from the `binary-builder` repo; for those binaries, you will usually need to change the recipe rather than the concourse task.

For the list of currently supported binaries, check out our `dependency-builds` [pipeline](https://buildpacks.ci.cf-app.com/teams/main/pipelines/dependency-builds).

The concourse task that orchestrates the building is `buildpacks-ci/tasks/build-binary/build.sh`; many of the recipes are in [binary-builder](https://github.com/cloudfoundry/binary-builder). The task is stack-agnostic — the stack image is passed in by the pipeline via the `image:` key on the task step.

To test these changes locally, you can execute the concourse task for it, but point to local changes. For instance:

```
$ cd buildpacks-ci
$ STACK=cflinuxfs4 fly -t buildpacks e -c tasks/build-binary/build.yml -j dependency-builds/build-r-3.4.X -i buildpacks-ci=.
```

For binaries that use recipes in `binary-builder`, you can also test in Docker. For instance:

```
$ docker run -w /binary-builder -v `pwd`:/binary-builder -it cloudfoundry/cflinuxfs2:ruby-2.2.4 ./bin/binary-builder --name=ruby --version=2.2.3 --md5=150a5efc5f5d8a8011f30aa2594a7654
$ ls
ruby-2.2.3-linux-x64.tgz
```

# Buildpack Repositories Guide

`buildpacks-ci` pipelines and tasks refer to many other repositories. These repos are where the buildpack team and others develop buildpacks and related artifacts.

## Officially-supported Buildpacks

Each officially-supported buildpack has a `develop` and a `master` branch.

Active development happens on `develop`. Despite our best efforts, `develop` will sometimes be unstable and is not production-ready.

Our release branch is `master`. This is stable and only updated with new buildpack releases.

* [binary-buildpack](https://github.com/cloudfoundry/binary-buildpack)
* [go-buildpack](https://github.com/cloudfoundry/go-buildpack)
* [nodejs-buildpack](https://github.com/cloudfoundry/nodejs-buildpack)
* [php-buildpack](https://github.com/cloudfoundry/php-buildpack)
* [python-buildpack](https://github.com/cloudfoundry/python-buildpack)
* [ruby-buildpack](https://github.com/cloudfoundry/ruby-buildpack)
* [dotnet-core-buildpack](https://github.com/cloudfoundry/dotnet-core-buildpack)
* [staticfile-buildpack](https://github.com/cloudfoundry/static-buildpack)

## Tooling for Development and Runtime

* [buildpack-packager](https://github.com/cloudfoundry/buildpack-packager)   Builds cached and uncached buildpacks
* [machete](https://github.com/cloudfoundry/machete)           Buildpack integration testing framework.
* [compile-extensions](https://github.com/cloudfoundry/compile-extensions) Suite of utility scripts used in buildpacks at runtime
* [libbuildpack](https://github.com/cloudfoundry/libbuildpack) Library used for writing buildpacks in Golang
* [binary-builder](https://github.com/cloudfoundry/binary-builder)           Builds binaries against specified rootfs
* [brats](https://github.com/cloudfoundry/brats) Buildpack Runtime Acceptance Test Suite, a collection of smoke tests

## BOSH Releases

BOSH releases are used in the assembly of [`cf-release`](https://github.com/cloudfoundry/cf-release).

* [go-buildpack-release](https://github.com/cloudfoundry/go-buildpack-release)
* [ruby-buildpack-release](https://github.com/cloudfoundry/ruby-buildpack-release)
* [python-buildpack-release](https://github.com/cloudfoundry/python-buildpack-release)
* [php-buildpack-release](https://github.com/cloudfoundry/php-buildpack-release)
* [nodejs-buildpack-release](https://github.com/cloudfoundry/nodejs-buildpack-release)
* [staticfile-buildpack-release](https://github.com/cloudfoundry/staticfile-buildpack-release)
* [binary-buildpack-release](https://github.com/cloudfoundry/binary-buildpack-release)
* [java-offline-buildpack-release](https://github.com/cloudfoundry/java-offline-buildpack-release)
* [java-buildpack-release](https://github.com/cloudfoundry/java-buildpack-release)
* [dotnet-core-buildpack-release](https://github.com/cloudfoundry/dotnet-core-buildpack-release)

## Experimental or unsupported

### Buildpacks

These buildpacks are possible candidates for promotion, or experimental architecture explorations.

* [hwc-buildpack](https://github.com/cloudfoundry/hwc-buildpack)
* [hwc-buildpack-release](https://github.com/cloudfoundry/hwc-buildpack-release)

### Tools

* [concourse-filter](https://github.com/pivotal-cf-experimental/concourse-filter) Redacts credentials from Concourse logs
* [new_version_resource](https://github.com/pivotal-cf-experimental/new_version_resource) Concourse resource to track dependency versions by scraping webpages

## Private Repos

Some repositories are private for historical or security reasons. We list them for completeness.

* [deployments-buildpacks](https://github.com/pivotal-cf/deployments-buildpacks) See repository README.
* [buildpacks-ci-robots](https://github.com/pivotal-cf/buildpacks-ci-robots) See repository README.

buildpacks-ci

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README