KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #typescript #developer-tools #claude-code #ai-agent #codex #deepseek-harness #ai-agents #cli #react #mcp

kaniko

★ 346 Open GitHub ↗

Build Container Images In Kubernetes

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

# kaniko - Build Images In Kubernetes

[![Unit tests](https://github.com/osscontainertools/kaniko/actions/workflows/unit-tests.yaml/badge.svg)](https://github.com/osscontainertools/kaniko/actions/workflows/unit-tests.yaml)
[![Integration tests](https://github.com/osscontainertools/kaniko/actions/workflows/integration-tests.yaml/badge.svg)](https://github.com/osscontainertools/kaniko/actions/workflows/integration-tests.yaml)
[![Build images](https://github.com/osscontainertools/kaniko/actions/workflows/images.yaml/badge.svg)](https://github.com/osscontainertools/kaniko/actions/workflows/images.yaml)
[![codecov](https://codecov.io/gh/osscontainertools/kaniko/graph/badge.svg)](https://codecov.io/gh/osscontainertools/kaniko)

![kaniko logo](https://raw.githubusercontent.com/osscontainertools/kaniko/main/logo/Kaniko-Logo.png)

kaniko is a tool to build container images from a Dockerfile, inside a container
or Kubernetes cluster.

> [!IMPORTANT]
> This is a supported replacement of the original `GoogleContainerTools/kaniko`
> repository, which was archived in June of 2025.
> The focus of this fork is to keep dependencies up-to-date, fix bugs and improve performance.
> The images are available on docker hub [martizih/kaniko](https://hub.docker.com/r/martizih/kaniko).
> If you are new here you can refer to our [Changelog Overview](./CHANGELOG_OVERVIEW.md) for the main differences to Google's v1.24.0 release.

kaniko doesn't depend on a Docker daemon and executes each command within a
Dockerfile completely in userspace. This enables building container images in
environments that can't easily or securely run a Docker daemon, such as a
standard Kubernetes cluster.

kaniko is meant to be run as an image: `ghcr.io/osscontainertools/kaniko:latest`. We do **not** recommend
running the kaniko executor binary in another image, as it might not work as you
expect - see [Known Issues](#known-issues).

**Table of Contents** _generated with
[DocToc](https://github.com/thlorenz/doctoc)_

- [kaniko - Build Images In Kubernetes](#kaniko---build-images-in-kubernetes)
  - [Community](#community)
  - [Sponsorships](#sponsorships)
    - [Corporations](#corporations)
    - [Individuals](#individuals)
  - [Alternatives](#alternatives)
  - [Releases](#releases)
  - [How does kaniko work?](#how-does-kaniko-work)
  - [Known Issues](#known-issues)
  - [Demo](#demo)
  - [Tutorial](#tutorial)
  - [Using kaniko](#using-kaniko)
    - [kaniko Build Contexts](#kaniko-build-contexts)
    - [Using Azure Blob Storage](#using-azure-blob-storage)
    - [Using Private Git Repository](#using-private-git-repository)
    - [Using Standard Input](#using-standard-input)
    - [Running kaniko](#running-kaniko)
      - [Running kaniko in a Kubernetes cluster](#running-kaniko-in-a-kubernetes-cluster)
        - [Kubernetes secret](#kubernetes-secret)
      - [Running kaniko in gVisor](#running-kaniko-in-gvisor)
      - [Running kaniko in Google Cloud Build](#running-kaniko-in-google-cloud-build)
      - [Running kaniko in Docker](#running-kaniko-in-docker)
    - [Bootstrapping Kaniko](#bootstrapping-kaniko)
    - [Caching](#caching)
      - [Caching Layers](#caching-layers)
      - [Caching Base Images](#caching-base-images)
    - [Pushing to Different Registries](#pushing-to-different-registries)
    - [Subcommands](#subcommands)
      - [Subcommand `login`](#subcommand-login)
      - [Subcommand `push`](#subcommand-push)
    - [Additional Flags](#additional-flags)
      - [Flag `--build-arg`](#flag---build-arg)
      - [Flag `--cache`](#flag---cache)
      - [Flag `--cache-dir`](#flag---cache-dir)
      - [Flag `--cache-repo`](#flag---cache-repo)
      - [Flag `--cache-copy-layers`](#flag---cache-copy-layers)
      - [Flag `--cache-run-layers`](#flag---cache-run-layers)
      - [Flag `--cache-ttl`](#flag---cache-ttl)
      - [Flag `--pre-cleanup`](#flag---pre-cleanup)
      - [Flag `--cleanup`](#flag---cleanup)
      - [Flag `--compression`](#flag---compression)
      - [Flag `--compression-level`](#flag---compression-level)
      - [Flag `--image-format`](#flag---image-format)
      - [Flag `--compressed-caching`](#flag---compressed-caching)
      - [Flag `--context-sub-path`](#flag---context-sub-path)
      - [Flag `--credential-helpers`](#flag---credential-helpers)
      - [Flag `--custom-platform`](#flag---custom-platform)
      - [Flag `--digest-file`](#flag---digest-file)
      - [Flag `--dockerfile`](#flag---dockerfile)
      - [Flag `--dryrun`](#flag---dryrun)
      - [Flag `--force`](#flag---force)
      - [Flag `--git`](#flag---git)
      - [Flag `--image-name-with-digest-file`](#flag---image-name-with-digest-file)
      - [Flag `--image-name-tag-with-digest-file`](#flag---image-name-tag-with-digest-file)
      - [Flag `--insecure`](#flag---insecure)
      - [Flag `--insecure-pull`](#flag---insecure-pull)
      - [Flag `--insecure-registry`](#flag---insecure-registry)
      - [Flag `--kaniko-dir`](#flag---kaniko-dir)
      - [Flag `--label`](#flag---label)
      - [Flag `--annotation`](#flag---annotation)
      - [Flag `--log-format`](#flag---log-format)
      - [Flag `--log-timestamp`](#flag---log-timestamp)
      - [Flag `--materialize`](#flag---materialize)
      - [Flag `--no-push`](#flag---no-push)
      - [Flag `--no-push-cache`](#flag---no-push-cache)
      - [Flag `--oci-layout-path`](#flag---oci-layout-path)
      - [Flag `--preserve-context`](#flag---preserve-context)
      - [Flag `--push-ignore-immutable-tag-errors`](#flag---push-ignore-immutable-tag-errors)
      - [Flag `--push-retry`](#flag---push-retry)
      - [Flag `--registry-certificate`](#flag---registry-certificate)
      - [Flag `--registry-client-cert`](#flag---registry-client-cert)
      - [Flag `--registry-map`](#flag---registry-map)
      - [Flag `--registry-mirror`](#flag---registry-mirror)
      - [Flag `--skip-default-registry-fallback`](#flag---skip-default-registry-fallback)
      - [Flag `--reproducible`](#flag---reproducible)
      - [Flag `--secret`](#flag---secret)
      - [Flag `--single-snapshot`](#flag---single-snapshot)
      - [Flag `--skip-push-permission-check`](#flag---skip-push-permission-check)
      - [Flag `--skip-tls-verify`](#flag---skip-tls-verify)
      - [Flag `--skip-tls-verify-pull`](#flag---skip-tls-verify-pull)
      - [Flag `--skip-tls-verify-registry`](#flag---skip-tls-verify-registry)
      - [Flag `--snapshot-mode`](#flag---snapshot-mode)
      - [Flag `--tar-path`](#flag---tar-path)
      - [Flag `--target`](#flag---target)
      - [Flag `--use-new-run`](#flag---use-new-run)
      - [Flag `--verbosity`](#flag---verbosity)
      - [Flag `--ignore-var-run`](#flag---ignore-var-run)
      - [Flag `--ignore-path`](#flag---ignore-path)
      - [Flag `--image-fs-extract-retry`](#flag---image-fs-extract-retry)
      - [Flag `--image-download-retry`](#flag---image-download-retry)
    - [Feature Flags](#feature-flags)
      - [Profiles](#profiles)
      - [Flag `FF_KANIKO_COPY_AS_ROOT`](#flag-ff_kaniko_copy_as_root)
      - [Flag `FF_KANIKO_IGNORE_CACHED_MANIFEST`](#flag-ff_kaniko_ignore_cached_manifest)
      - [Flag `FF_KANIKO_RUN_MOUNT_BIND`](#flag-ff_kaniko_run_mount_bind)
      - [Flag `FF_KANIKO_DISABLE_HTTP2`](#flag-ff_kaniko_disable_http2)
      - [Flag `FF_KANIKO_OCI_WARMER`](#flag-ff_kaniko_oci_warmer)
      - [Flag `FF_KANIKO_RUN_VIA_TINI`](#flag-ff_kaniko_run_via_tini)
      - [Flag `FF_KANIKO_COPY_CHMOD_ON_IMPLICIT_DIRS`](#flag-ff_kaniko_copy_chmod_on_implicit_dirs)
      - [Flag `FF_KANIKO_CHOWN_ON_IMPLICIT_DIRS`](#flag-ff_kaniko_chown_on_implicit_dirs)
      - [Flag `FF_KANIKO_CLEAN_KANIKO_DIR`](#flag-ff_kaniko_clean_kaniko_dir)
      - [Flag `FF_KANIKO_NO_PROPAGATE_ANNOTATIONS`](#flag-ff_kaniko_no_propagate_annotations)
      - [Flag `FF_KANIKO_OCI_SCRATCH_BASE`](#flag-ff_kaniko_oci_scratch_base)
      - [Flag `FF_KANIKO_VOLUME_SKIP_MKDIR`](#flag-ff_kaniko_volume_skip_mkdir)
      - [Flag `FF_KANIKO_PRESERVE_HARDLINKS`](#flag-ff_kaniko_preserve_hardlinks)
      - [Flag `FF_KANIKO_RELATIVE_LINK_TARGETS`](#flag-ff_kaniko_relative_link_targets)
      - [Flag `FF_KANIKO_SKIP_WRITE_WHITEOUTS`](#flag-ff_kaniko_skip_write_whiteouts)
      - [Flag `FF_KANIKO_BUILDKIT_ARG_ENV_PRECEDENCE`](#flag-ff_kaniko_buildkit_arg_env_precedence)
      - [Flag `FF_KANIKO_INFER_CROSS_STAGE_CACHE_KEY`](#flag-ff_kaniko_infer_cross_stage_cache_key)
      - [Flag `FF_KANIKO_CACHE_LOOKAHEAD`](#flag-ff_kaniko_cache_lookahead)
      - [Flag `FF_KANIKO_ROLLING_CACHE_KEY`](#flag-ff_kaniko_rolling_cache_key)
      - [Flag `FF_KANIKO_HASH_DIR_FRAMING`](#flag-ff_kaniko_hash_dir_framing)
      - [Flag `FF_KANIKO_CACHE_PROBE_AFTER_MISS`](#flag-ff_kaniko_cache_probe_after_miss)
      - [Flag `FF_KANIKO_WARMER_CACHE_LOCK`](#flag-ff_kaniko_warmer_cache_lock)
      - [Flag `FF_KANIKO_PRESERVE_MOUNTED_PATHS`](#flag-ff_kaniko_preserve_mounted_paths)
      - [Flag `FF_KANIKO_REPRODUCIBLE_PRESERVE_BASE_LAYERS`](#flag-ff_kaniko_reproducible_preserve_base_layers)
      - [Flag `FF_KANIKO_DEPRECATE_INTER_STAGE_RESTORE`](#flag-ff_kaniko_deprecate_inter_stage_restore)
      - [Flag `FF_KANIKO_SCOPED_DOCKERIGNORE`](#flag-ff_kaniko_scoped_dockerignore)
      - [Flag `FF_KANIKO_PRECOMPILE_DOCKERIGNORE`](#flag-ff_kaniko_precompile_dockerignore)
      - [Flag `FF_KANIKO_SKIP_RELABEL_RECOMPRESS`](#flag-ff_kaniko_skip_relabel_recompress)
      - [Flag `FF_KANIKO_SECUREJOIN_EXTRACTION`](#flag-ff_kaniko_securejoin_extraction)
      - [Flag `FF_KANIKO_RESOLVE_CACHE_KEY`](#flag-ff_kaniko_resolve_cache_key)
      - [Flag `FF_KANIKO_UNTAR_SKIP_ROOT`](#flag-ff_kaniko_untar_skip_root)
      - [Flag `FF_KANIKO_RUN_HONOR_GROUP`](#flag-ff_kaniko_run_honor_group)
      - [Flag `FF_KANIKO_EXPAND_HEREDOC`](#flag-ff_kaniko_expand_heredoc)
      - [Flag `FF_KANIKO_SKIP_CACHED_STAGES`](#flag-ff_kaniko_skip_cached_stages)
      - [Flag `FF_KANIKO_SHARED_BASE_CACHE`](#flag-ff_kaniko_shared_base_cache)
      - [Flag `FF_KANIKO_CROSS_REPO_MOUNT`](#flag-ff_kaniko_cross_repo_mount)
    - [Assertion Overrides](#assertion-overrides)
    - [Telemetry](#telemetry)
    - [Debug Image](#debug-image)
  - [Security](#security)
    - [Verifying Signed Kaniko Images](#verifying-signed-kaniko-images)
  - [Creating Multi-arch Container Manifests Using Kaniko and Manifest-tool](#creating-multi-arch-container-manifests-using-kaniko-and-manifest-tool)
    - [General Workflow](#general-workflow)
    - [Limitations and Pitfalls](#limitations-and-pitfalls)
    - [Example CI Pipeline (GitLab)](#example-ci-pipeline-gitlab)
      - [Building the Separate Container Images](#building-the-separate-container-images)
      - [Merging the Container Manifests](#merging-the-container-manifests)
      - [On the Note of Adding Versioned Tags](#on-the-note-of-adding-versioned-tags)
  - [Comparison with Other Tools](#comparison-with-other-tools)
  - [Limitations](#limitations)
    - [mtime and snapshotting](#mtime-and-snapshotting)
    - [Dockerfile commands `--chown` support](#dockerfile-commands---chown-support)
  - [References](#references)

## Community

If you are interested in contributing to kaniko, learn more from our
[development](DEVELOPMENT.md) and [contributing](CONTRIBUTING.md) guides.

For any community discussion [participate in open
issues](https://github.com/osscontainertools/kaniko/issues) or [file a new
issue](https://github.com/osscontainertools/kaniko/issues/new/choose).

Join our official chat on Matrix at [#kaniko:matrix.org](https://app.element.io/#/room/#kaniko:matrix.org). It has dedicated rooms for [#support@kaniko:matrix.org](https://app.element.io/#/room/#support@kaniko:matrix.org) and [#announcements@kaniko:matrix.org](https://app.element.io/#/room/#announcements@kaniko:matrix.org). We will also continue to monitor discussions in the [#kaniko on Kubernetes Slack](https://kubernetes.slack.com/messages/CQDCHGX7Y/).

## Sponsorships
We are grateful to the organizations and individuals who support our project.

### Corporations
[<img src="docs/sponsors/l3montree-logo.svg" alt="L3montree" width="250">](https://l3montree.com)

**[L3montree](https://l3montree.com)**

We thank L3montree for their generous support and commitment to open-source sustainability.

[<img src="docs/sponsors/siemens-logo.svg" alt="Siemens" width="250">](https://opensource.siemens.com)

**[Siemens](https://opensource.siemens.com)**

We thank Siemens for their generous support, and especially the team behind [opensource.siemens.com](https://opensource.siemens.com).

### Individuals
<table>
<tr>
<td align="center"><a href="https://github.com/Sped0n"><img src="https://github.com/Sped0n.png?size=80" width="80" alt="Sped0n"> Sped0n</a></td>
<td align="center"><a href="https://github.com/bootc"><img src="https://github.com/bootc.png?size=80" width="80" alt="bootc"> bootc</a></td>
</tr>
</table>

We are grateful for your support.

## Alternatives

Chainguard forked kaniko and continues to maintain the project as https://github.com/chainguard-dev/kaniko.
Chainguard is a company founded by the [original authors](https://github.com/chainguard-dev/kaniko?tab=readme-ov-file#history-and-status) of kaniko and hence it is a project dear to their heart.
Their focus is to keep dependencies up to date and patch security issues, keeping kaniko more or less as-is feature wise. They do not release images publicly, only to chainguard customers. However, there is good guidance on how to build kaniko yourself from their source-only releases.

## Releases

kaniko releases are published as images on [ghcr.io/osscontainertools/kaniko](https://github.com/orgs/osscontainertools/packages/container/package/kaniko) and on Docker Hub as [martizih/kaniko](https://hub.docker.com/r/martizih/kaniko).

Release notes and source code archives are available on the [releases
section](https://github.com/osscontainertools/kaniko/releases). For the release cadence and feature flag graduation policy, see [docs/releases.md](docs/releases.md).

Images available from other vendors:

* [kaniko-build organization](https://github.com/kaniko-build)

## How does kaniko work?

The kaniko executor image is responsible for building an image from a Dockerfile
and pushing it to a registry. Within the executor image, we extract the
filesystem of the base image (the FROM image in the Dockerfile). We then execute
the commands in the Dockerfile, snapshotting the filesystem in userspace after
each one. After each command, we append a layer of changed files to the base
image (if there are any) and update image metadata.

## Known Issues

- kaniko does not support building Windows containers.
- Running kaniko in any Docker image other than the official kaniko image is not
  supported due to implementation details.
  - This includes copying the kaniko executables from the official image into
    another image (e.g. a Jenkins CI agent).
  - In particular, it cannot use chroot or bind-mount because its container must
    not require privilege, so it unpacks directly into its own container root
    and may overwrite anything already there.
- kaniko does not support the v1 Registry API
  ([Registry v1 API Deprecation](https://www.docker.com/blog/registry-v1-api-deprecation/))

## Demo

![Demo](https://raw.githubusercontent.com/osscontainertools/kaniko/main/docs/demo.gif)

## Tutorial

For a detailed example of kaniko with local storage, please refer to a
[getting started tutorial](./docs/tutorial.md).

Please see [References](#References) for more docs & video tutorials

## Using kaniko

To use kaniko to build and push an image for you, you will need:

1. A [build context](#kaniko-build-contexts), aka something to build
2. A [running instance of kaniko](#running-kaniko)

### kaniko Build Contexts

kaniko's build context is very similar to the build context you would send your
Docker daemon for an image build; it represents a directory containing a
Dockerfile which kaniko will use to build your image. For example, a `COPY`
command in your Dockerfile should refer to a file in the build context.

You will need to store your build context in a place that kaniko can access.
Right now, kaniko supports these storage solutions:

- GCS Bucket
- S3 Bucket
- Azure Blob Storage
- Local Directory
- Local Tar
- Standard Input
- Git Repository

_Note about Local Directory: this option refers to a directory within the kaniko
container. If you wish to use this option, you will need to mount in your build
context into the container as a directory._

_Note about Local Tar: this option refers to a tar gz file within the kaniko
container. If you wish to use this option, you will need to mount in your build
context into the container as a file._

_Note about Standard Input: the only Standard Input allowed by kaniko is in
`.tar.gz` format._

If using a GCS or S3 bucket, you will first need to create a compressed tar of
your build context and upload it to your bucket. Once running, kaniko will then
download and unpack the compressed tar of the build context before starting the
image build.

To create a compressed tar, you can run:

```shell
tar -C <path to build context> -zcvf context.tar.gz .
```

Then, copy over the compressed tar into your bucket. For example, we can copy
over the compressed tar to a GCS bucket with gsutil:

```shell
gsutil cp context.tar.gz gs://<bucket name>
```

When running kaniko, use the `--context` flag with the appropriate prefix to
specify the location of your build context:

| Source | Prefix | Example |
| ------------------ | --------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| Local Directory | dir://[path to a directory in the kaniko container] | `dir:///workspace` |
| Local Tar Gz | tar://[path to a .tar.gz in the kaniko container] | `tar:///path/to/context.tar.gz` |
| Standard Input | tar://[stdin] | `tar://stdin` |
| GCS Bucket | gs://[bucket name]/[path to .tar.gz] | `gs://kaniko-bucket/path/to/context.tar.gz` |
| S3 Bucket | s3://[bucket name]/[path to .tar.gz] | `s3://kaniko-bucket/path/to/context.tar.gz` |
| Azure Blob Storage | https://[account].[azureblobhostsuffix]/[container]/[path to .tar.gz] | `https://myaccount.blob.core.windows.net/container/path/to/context.tar.gz` |
| Git Repository | git://[repository url][#reference][#commit-id] | `git://github.com/acme/myproject.git#refs/heads/mybranch#<desired-commit-id>` |

If you don't specify a prefix, kaniko will assume a local directory. For
example, to use a GCS bucket called `kaniko-bucket`, you would pass in
`--context=gs://kaniko-bucket/path/to/context.tar.gz`.

### Using Azure Blob Storage

If you are using Azure Blob Storage for context file, you will need to pass
[Azure Storage Account Access Key](https://docs.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string?toc=%2fazure%2fstorage%2fblobs%2ftoc.json)
as an environment variable named `AZURE_STORAGE_ACCESS_KEY` through Kubernetes
Secrets

### Using Private Git Repository

You can use `Personal Access Tokens` for Build Contexts from Private
Repositories from
[GitHub](https://blog.github.com/2012-09-21-easier-builds-and-deployments-using-git-over-https-and-oauth/).

You can either pass this in as part of the git URL (e.g.,
`git://TOKEN@github.com/acme/myproject.git#refs/heads/mybranch`) or using the
environment variable `GIT_TOKEN`.

You can also pass `GIT_USERNAME` and `GIT_PASSWORD` (password being the token)
if you want to be explicit about the username.

### Using Standard Input

If running kaniko and using Standard Input build context, you will need to add
the docker or kubernetes `-i, --interactive` flag. Once running, kaniko will
then get the data from `STDIN` and create the build context as a compressed tar.
It will then unpack the compressed tar of the build context before starting the
image build. If no data is piped during the interactive run, you will need to
send the EOF signal by yourself by pressing `Ctrl+D`.

Complete example of how to interactively run kaniko with `.tar.gz` Standard
Input data, using docker:

```shell
echo -e 'FROM alpine \nRUN echo "created from standard input"' > Dockerfile | tar -cf - Dockerfile | gzip -9 | docker run \
 --interactive -v $(pwd):/workspace ghcr.io/osscontainertools/kaniko:latest \
 --context tar://stdin \
 --destination=<YOUR-REGISTRY>/$project/$image:$tag>
```

Complete example of how to interactively run kaniko with `.tar.gz` Standard
Input data, using Kubernetes command line with a temporary container and
completely dockerless:

```shell
echo -e 'FROM alpine \nRUN echo "created from standard input"' > Dockerfile | tar -cf - Dockerfile | gzip -9 | kubectl run kaniko \
--rm --stdin=true \
--image=ghcr.io/osscontainertools/kaniko:latest --restart=Never \
--overrides='{
 "apiVersion": "v1",
 "spec": {
 "containers": [
 {
 "name": "kaniko",
 "image": "ghcr.io/osscontainertools/kaniko:latest",
 "stdin": true,
 "stdinOnce": true,
 "args": [
 "--dockerfile=Dockerfile",
 "--context=tar://stdin",
 "--destination<YOUR-REGISTRY>/<YOUR-REPO>/my-image"
 ],
 "volumeMounts": [
 {
 "name": "cabundle",
 "mountPath": "/kaniko/ssl/certs/"
 },
 {
 "name": "docker-config",
 "mountPath": "/kaniko/.docker/"
 }
 ]
 }
 ],
 "volumes": [
 {
 "name": "cabundle",
 "configMap": {
 "name": "cabundle"
 }
 },
 {
 "name": "docker-config",
 "configMap": {
 "name": "docker-config"
 }
 }
 ]
 }
}'
```

### Running kaniko

There are several different ways to deploy and run kaniko:

- [In a Kubernetes cluster](#running-kaniko-in-a-kubernetes-cluster)
- [In gVisor](#running-kaniko-in-gvisor)
- [In Google Cloud Build](#running-kaniko-in-google-cloud-build)
- [In Docker](#running-kaniko-in-docker)

#### Running kaniko in a Kubernetes cluster

Requirements:

- Standard Kubernetes cluster (e.g. using
  [GKE](https://cloud.google.com/kubernetes-engine/))
- [Kubernetes Secret](#kubernetes-secret)
- A [build context](#kaniko-build-contexts)

##### Kubernetes secret

To run kaniko in a Kubernetes cluster, you will need a standard running
Kubernetes cluster and a Kubernetes secret, which contains the auth required to
push the final image.

To create a secret to authenticate to Google Cloud Registry, follow these steps:

1. Create a service account in the Google Cloud Console project you want to push
   the final image to with `Storage Admin` permissions.
2. Download a JSON key for this service account
3. Rename the key to `kaniko-secret.json`
4. To create the secret, run:

```shell
kubectl create secret generic kaniko-secret --from-file=<path to kaniko-secret.json>
```

_Note: If using a GCS bucket in the same GCP project as a build context, this
service account should now also have permissions to read from that bucket._

The Kubernetes Pod spec should look similar to this, with the args parameters
filled in:

```yaml
apiVersion: v1
kind: Pod
metadata:
 name: kaniko
spec:
 containers:
 - name: kaniko
 image: ghcr.io/osscontainertools/kaniko:latest
 args:
 - "--dockerfile=<path to Dockerfile within the build context>"
 - "--context=gs://<GCS bucket>/<path to .tar.gz>"
 - "--destination=<<YOUR-REGISTRY>/$PROJECT/$IMAGE:$TAG>"
 volumeMounts:
 - name: kaniko-secret
 mountPath: /secret
 env:
 - name: GOOGLE_APPLICATION_CREDENTIALS
 value: /secret/kaniko-secret.json
 restartPolicy: Never
 volumes:
 - name: kaniko-secret
 secret:
 secretName: kaniko-secret
```

This example pulls the build context from a GCS bucket. To use a local directory
build context, you could consider using configMaps to mount in small build
contexts.

#### Running kaniko in gVisor

Running kaniko in [gVisor](https://github.com/google/gvisor) provides an
additional security boundary. You will need to add the `--force` flag to run
kaniko in gVisor, since currently there isn't a way to determine whether or not
a container is running in gVisor.

```shell
docker run --runtime=runsc -v $(pwd):/workspace -v ~/.config:/root/.config \
ghcr.io/osscontainertools/kaniko:latest \
--dockerfile=<path to Dockerfile> --context=/workspace \
--destination=<YOUR-REGISTRY>/<YOUR-REPO>/my-image --force
```

We pass in `--runtime=runsc` to use gVisor. This example mounts the current
directory to `/workspace` for the build context and the `~/.config` directory
for GCR credentials.

#### Running kaniko in Google Cloud Build

Requirements:

- A [build context](#kaniko-build-contexts)

To run kaniko in GCB, add it to your build config as a build step:

```yaml
steps:
 - name: ghcr.io/osscontainertools/kaniko:latest
 args:
 [
 "--dockerfile=<path to Dockerfile within the build context>",
 "--context=dir://<path to build context>",
 "--destination=<<YOUR-REGISTRY>/$PROJECT/$IMAGE:$TAG>",
 ]
```

kaniko will build and push the final image in this build step.

#### Running kaniko in Docker

Requirements:

- [Docker](https://docs.docker.com/install/)

We can run the kaniko executor image locally in a Docker daemon to build and
push an image from a Dockerfile.

For example, when using gcloud and GCR you could run kaniko as follows:

```shell
docker run \
    -v "$HOME"/.config/gcloud:/root/.config/gcloud \
    -v /path/to/context:/workspace \
    ghcr.io/osscontainertools/kaniko:latest \
    --dockerfile /workspace/Dockerfile \
    --destination "gcr.io/$PROJECT_ID/$IMAGE_NAME:$TAG" \
    --context dir:///workspace/
```

There is also a utility script [`run_in_docker.sh`](./run_in_docker.sh) that can
be used as follows:

```shell
./run_in_docker.sh <path to Dockerfile> <path to build context> <destination of final image>
```

_NOTE: `run_in_docker.sh` expects a path to a Dockerfile relative to the
absolute path of the build context._

An example run, specifying the Dockerfile in the container directory
`/workspace`, the build context in the local directory
`/home/user/kaniko-project`, and a Google Container Registry as a remote image
destination:

```shell
./run_in_docker.sh /workspace/Dockerfile /home/user/kaniko-project gcr.io/$PROJECT_ID/$TAG
```

### Bootstrapping Kaniko
Kaniko's approach to buliding docker images is unique. It doesn't build up and snapshot an image in a separate container, instead it will build up and snapshot the image in the container where kaniko is installed. The benefit is that kaniko can run without any privileges, because it's not actually using any containerization technologies, the downside is that sometimes builds can yield surprising results - bootstrapping kaniko image with kaniko builder is one such case.

The good news is, it's all technically possible, but it's a bit more involved than building other images.

The first problem we face is that the kaniko binaries are installed in `/kaniko` directory. Which means that this directory must also be ignored during snapshots, lest we would leak our build tool into any image we produce. But this also means that kaniko by default can't build a kaniko image with the binaries installed in `/kaniko` directory. Luckily there is an override for that, that allows us to move all the binaries to a different location before the build ie. `KANIKO_DIR=/kaniko2`.

The second problem only affects build using the `debug` image, ie. gitlab-runner. The shell that is spawned in the debug image is in `/busybox`, similarly we can't snapshot files in that directory. Unfortunately there is no override to move those binaries. But with a bit creativity we can create a bootstrap image that has the shell installed into a different location ie. `/busybox2` and then use that bootstrap image to build the actual new debug image.

```yaml
bootstrap:
  extends:
    - .build
  image:
    name: gcr.io/kaniko-project/executor:v1.24.0-debug
    entrypoint: [""]
  needs: []
  variables:
    IMAGE: ${CI_REGISTRY_IMAGE}/bootstrap:latest
    EXTRA_ARGS: >-
      --build-arg=TARGETARCH=amd64
      --build-arg=TARGETOS=linux
      --target=kaniko-debug-2
    KANIKO_DIR: /kaniko2

build:
  extends:
    - .build
  image:
    name: ${CI_REGISTRY_IMAGE}/bootstrap:latest
    entrypoint: [""]
  needs: [bootstrap]
  variables:
    IMAGE: ${CI_REGISTRY_IMAGE}/kaniko:latest
    EXTRA_ARGS: >-
      --build-arg=TARGETARCH=amd64
      --build-arg=TARGETOS=linux
      --target=kaniko-debug
    KANIKO_DIR: /kaniko2
```
This is just an illustrative extract, please find the full [.gitlab-ci.yml](./docs/bootstrap.gitlab-ci.yml) here.

With this two step approach we can now indeed bootstrap kaniko in kaniko. However, it is not really necessary to rebuild that intermediate bootstrap image every time, we can reuse it from a different build. Hence I provide a dedicated image `ghcr.io/osscontainertools/kaniko:bootstrap` that can be used for that purpose.

### Caching

#### Caching Layers

kaniko can cache layers created by `RUN`(configured by flag
`--cache-run-layers`) and `COPY` (configured by flag `--cache-copy-layers`)
commands in a remote repository. Before executing a command, kaniko checks the
cache for the layer. If it exists, kaniko will pull and extract the cached layer
instead of executing the command. If not, kaniko will execute the command and
then push the newly created layer to the cache.

Note that kaniko cannot read layers from the cache after a cache miss: once a
layer has not been found in the cache, all subsequent layers are built locally
without consulting the cache.

Users can opt into caching by setting the `--cache=true` flag. A remote
repository for storing cached layers can be provided via the `--cache-repo`
flag. If this flag isn't provided, a cached repo will be inferred from the
`--destination` provided.

#### Caching Base Images

kaniko can cache images in a local directory that can be volume mounted into the
kaniko pod. To do so, the cache must first be populated, as it is read-only. We
provide a kaniko cache warming image at `gcr.io/kaniko-project/warmer`:

```shell
docker run -v $(pwd):/workspace ghcr.io/osscontainertools/kaniko:warmer --cache-dir=/workspace/cache --image=<image to cache> --image=<another image to cache>
docker run -v $(pwd):/workspace ghcr.io/osscontainertools/kaniko:warmer --cache-dir=/workspace/cache --dockerfile=<path to dockerfile>
docker run -v $(pwd):/workspace ghcr.io/osscontainertools/kaniko:warmer --cache-dir=/workspace/cache --dockerfile=<path to dockerfile> --build-arg version=1.19
```

`--image` can be specified for any number of desired images. `--dockerfile` can
be specified for the path of dockerfile for cache.These command will combined to
cache those images by digest in a local directory named `cache`. Once the cache
is populated, caching is opted into with the same `--cache=true` flag as above.
The location of the local cache is provided via the `--cache-dir` flag,
defaulting to `/cache` as with the cache warmer. See the `examples` directory
for how to use with kubernetes clusters and persistent cache volumes.

### Pushing to Different Registries

For registry-specific setup instructions (Docker Hub, GCR, ECR, ACR, JFrog,
registry mirrors and maps) see **[docs/registries.md](docs/registries.md)**.

### Subcommands

In addition to the default build-and-push flow, the `executor` binary exposes a small set of subcommands.

#### Subcommand `login`

`executor login <registry>` stores registry credentials in the Docker config file at `$DOCKER_CONFIG/config.json` (the executor image sets `DOCKER_CONFIG=/kaniko/.docker/`), so subsequent `executor` invocations can authenticate to the registry without a credential helper.

Flags:

- `-u, --username <user>` — username for the registry (required).
- `-p, --password <pass>` — password or token; mutually exclusive with `--password-stdin`.
- `--password-stdin` — read the password from standard input; useful for piping a secret without exposing it in the process list.

#### Subcommand `push`

`executor push <path> --destination <ref>` reads a pre-built image and pushes it to one or more registries, skipping the build entirely. The path may point at a docker-save format tarball produced by [`--tar-path`](#flag---tar-path) or at an OCI image layout directory produced by [`--oci-layout-path`](#flag---oci-layout-path). All registry, auth, retry, and digest-file flags are the same as on the build command. See the canonical workflow under [`--tar-path`](#flag---tar-path).

The artifact must contain exactly one image. Multi-image tarballs and indexes are not supported.

### Additional Flags

#### Flag `--build-arg`

This flag allows you to pass in ARG values at build time, similarly to Docker.
You can set it multiple times for multiple arguments.

Note that passing values that contain spaces is not natively supported - you
need to ensure that the IFS is set to null before your executor command. You can
set this by adding `export IFS=''` before your executor call. See the following
example

```bash
export IFS=''
/kaniko/executor --build-arg "MY_VAR='value with spaces'" ...
```

#### Flag `--cache`

Set this flag as `--cache=true` to opt into caching with kaniko.

#### Flag `--cache-dir`

Set this flag to specify a local directory cache for base images. Defaults to
`/cache`.

_This flag must be used in conjunction with the `--cache=true` flag._

#### Flag `--cache-repo`

Set this flag to specify a remote repository that will be used to store cached
layers.

If this flag is not provided, a cache repo will be inferred from the
`--destination` flag. If `--destination=gcr.io/kaniko-project/test`, then cached
layers will be stored in `gcr.io/kaniko-project/test/cache`.

_This flag must be used in conjunction with the `--cache=true` flag._

#### Flag `--cache-copy-layers`

Set this flag to cache copy layers.

#### Flag `--cache-run-layers`

Set this flag to cache run layers (default=true).

#### Flag `--cache-ttl`

Cache timeout in hours. Defaults to two weeks.

#### Flag `--pre-cleanup`

Set this flag to clean the filesystem before the build.
ie. in order to support custom built kaniko images.

Defaults to `false`. Can also be set via the `KANIKO_PRE_CLEANUP` environment variable.

#### Flag `--cleanup`

Set this flag to clean the filesystem and kaniko's working directory at the end of the build.

Defaults to `false`. Can also be set via the `KANIKO_CLEANUP` environment variable.

#### Flag `--compression`

Use this flag to select the compression algorithm `[gzip, zstd]`. Defaults to `gzip`.

#### Flag `--compression-level`

Use this flag to select the compression level. Defaults to `-1` (no compression)

#### Flag `--image-format`

Use this flag to select the output image media type `[docker, oci]`. `docker` writes a Docker schema2 manifest, `oci` writes an OCI image manifest. When unset, kaniko inherits the format of the base image.

#### Flag `--compressed-caching`

Set this to false in order to prevent tar compression for cached layers. This
will increase the runtime of the build, but decrease the memory usage especially
for large builds. Try to use `--compressed-caching=false` if your build fails
with an out of memory error. Defaults to true.

#### Flag `--context-sub-path`

Set a sub path within the given `--context`.

Its particularly useful when your context is, for example, a git repository, and
you want to build one of its subfolders instead of the root folder.

#### Flag `--credential-helpers`

Use these credential helpers automatically, select from (env, google, ecr, acr, gitlab). Set it repeatedly for multiple helpers, defaults to all, set it to empty string to deactivate.

#### Flag `--custom-platform`

Allows to build with another default platform than the host, similarly to docker
build --platform xxx the value has to be on the form
`--custom-platform=linux/arm`, with acceptable values listed here:
[GOOS/GOARCH](https://gist.github.com/asukakenji/f15ba7e588ac42795f421b48b8aede63).

It's also possible specifying CPU variants adding it as a third parameter (like
`--custom-platform=linux/arm/v5`). Currently CPU variants are only known to be
used for the ARM architecture as listed here:
[GOARM](https://go.dev/wiki/GoArm#supported-architectures)

_The resulting images cannot provide any metadata about CPU variant due to a
limitation of the OCI-image specification._

_This is not virtualization and cannot help to build an architecture not
natively supported by the build host. This is used to build i386 on an amd64
Host for example, or arm32 on an arm64 host._

#### Flag `--digest-file`

Set this flag to specify a file in the container. This file will receive the
digest of a built image. This can be used to automatically track the exact image
built by kaniko.

For example, setting the flag to `--digest-file=/dev/termination-log` will write
the digest to that file, which is picked up by Kubernetes automatically as the
`{{.state.terminated.message}}` of the container.

#### Flag `--dockerfile`

Path to the dockerfile to be built. (default "Dockerfile")

#### Flag `--dryrun`

Instead of building the docker image, just print a plan of what kaniko would do.

#### Flag `--force`

Force building outside of a container

#### Flag `--git`

Branch to clone if build context is a git repository (default
branch=,single-branch=false,depth=0,recurse-submodules=false,insecure-skip-tls=false)

#### Flag `--image-name-with-digest-file`

Specify a file to save the image name w/ digest of the built image to.

#### Flag `--image-name-tag-with-digest-file`

Specify a file to save the image name w/ image tag and digest of the built image
to.

#### Flag `--insecure`

Set this flag if you want to push images to a plain HTTP registry. It is
supposed to be used for testing purposes only and should not be used in
production!

#### Flag `--insecure-pull`

Set this flag if you want to pull images from a plain HTTP registry. It is
supposed to be used for testing purposes only and should not be used in
production!

#### Flag `--insecure-registry`

You can set `--insecure-registry <registry-name>` to use plain HTTP requests
when accessing the specified registry. It is supposed to be used for testing
purposes only and should not be used in production! You can set it multiple
times for multiple registries.

#### Flag `--kaniko-dir`

Set this flag as `--kaniko-dir /not-kaniko` to move the kaniko binaries to `/not-kaniko` before the build starts. This is helpful in [Bootstrapping Kaniko](#bootstrapping-kaniko).
Will be deprecated in `v1.28.0`. Use the env variable `KANIKO_DIR` instead.

#### Flag `--label`

Set this flag as `--label key=value` to set some metadata to the final image.
This is equivalent as using the `LABEL` within the Dockerfile.

#### Flag `--annotation`

Set this flag as `--annotation key=value` to set some metadata to the final image.
[Annotation levels](https://docs.docker.com/build/metadata/annotations/#specify-annotation-level)
are currently not supported and it's always the manifest that's
annotated.

#### Flag `--log-format`

Set this flag as `--log-format=<text|color|json>` to set the log format.
Defaults to `color`.

#### Flag `--log-timestamp`

Set this flag as `--log-timestamp=<true|false>` to add timestamps to
`<text|color>` log format. Defaults to `false`.

#### Flag `--materialize`

Set this boolean flag to `true` if you want kaniko to ensure that the filesystem is in a well-defined state after the build finishes. If you have a 100% cache-hitrate kaniko can skip unpacking files to the filesystem as it is superfluous if the goal is to simply build an image and push it to registry, but this also means that state of the filesystem after the build depends on whether that optimization was possible or not. This option disables this optimization entirely making sure that the filesystem is always well-defined.

This is useful if you use kaniko not to build a docker image, but to initialize your environment.

Defaults to `false`

#### Flag `--no-push`

Set this flag if you only want to build the image, without pushing to a
registry. This can also be defined through `KANIKO_NO_PUSH` environment
variable.

NOTE: this will still push cache layers to the repo, to disable pushing cache layers use `--no-push-cache`

#### Flag `--no-push-cache`

Set this flag if you do not want to push cache layers to a
registry.  Can be used in addition to `--no-push` to push no layers to a registry.

#### Flag `--oci-layout-path`

Set this flag to specify a directory in the container where the OCI image layout
of a built image will be placed. This can be used to automatically track the
exact image built by kaniko.

For example, to surface the image digest built in a
[Tekton task](https://github.com/tektoncd/pipeline/blob/v0.6.0/docs/resources.md#surfacing-the-image-digest-built-in-a-task),
this flag should be set to match the image resource `outputImageDir`.

_Note: Depending on the built image, the media type of the image manifest might
be either `application/vnd.oci.image.manifest.v1+json` or
`application/vnd.docker.distribution.manifest.v2+json`._

#### Flag `--preserve-context`

Set this boolean flag to `true` if you want kaniko to restore the build-context for multi-stage builds.
If set, kaniko will take a snapshot of the full filesystem before it starts building to later restore to that state. If combined with the `--cleanup` flag it will also restore the state after cleanup. If combined with `--pre-cleanup` it will **not** restore the state in between stages.

This is useful if you want to pass in secrets via files or if you want to execute commands after the build completes.

It will only take the snapshot if we are building a multistage image or if we plan to cleanup the filesystem either before or after the build.

Defaults to `false`. Can also be set via `KANIKO_PRESERVE_CONTEXT` environment variable.

#### Flag `--push-ignore-immutable-tag-errors`

Set this boolean flag to `true` if you want the Kaniko process to exit with
success when a push error related to tag immutability occurs.

This is useful for example if you have parallel builds pushing the same tag
and do not care which one actually succeeds.

Defaults to `false`.

#### Flag `--push-retry`

Set this flag to the number of retries that should happen for the push of an
image to a remote destination. Defaults to `0`.

#### Flag `--registry-certificate`

Set this flag to provide a certificate for TLS communication with a given
registry.

Expected format is `my.registry.url=/path/to/the/certificate.cert`

#### Flag `--registry-client-cert`

Set this flag to provide a certificate/key pair for mutual TLS (mTLS)
communication with a given
[registry that requires mTLS](https://docs.docker.com/engine/security/certificates/)
for authentication.

Expected format is
`my.registry.url=/path/to/client/cert.crt,/path/to/client/key.key`

#### Flag `--registry-map`

Set this flag if you want to remap registries references. Useful for air gap
environment for example. You can use this flag more than once, if you want to
set multiple mirrors for a given registry. You can mention several remap in a
single flag too, separated by semi-colon. If an image is not found on the first
mirror, Kaniko will try the next mirror(s), and at the end fallback on the
original registry.

Registry maps can also be defined through `KANIKO_REGISTRY_MAP` environment
variable.

Expected format is
`original-registry=remapped-registry[;another-reg=another-remap[;...]]` for
example.

Note that you **can** specify a URL with scheme for this flag. Some valid options
are:

- `index.docker.io=mirror.gcr.io`
- `gcr.io=127.0.0.1`
- `quay.io=192.168.0.1:5000`
- `index.docker.io=docker-io.mirrors.corp.net;index.docker.io=mirror.gcr.io;gcr.io=127.0.0.1`
  will try `docker-io.mirrors.corp.net` then `mirror.gcr.io` for
  `index.docker.io` and `127.0.0.1` for `gcr.io`
- `docker.io=harbor.private.io/theproject`

#### Flag `--registry-mirror`

Set this flag if you want to use a registry mirror instead of the default
`index.docker.io`. You can use this flag more than once, if you want to set
multiple mirrors. If an image is not found on the first mirror, Kaniko will try
the next mirror(s), and at the end fallback on the default registry.

Mirror can also be defined through `KANIKO_REGISTRY_MIRROR` environment
variable.

Expected format is `mirror.gcr.io` or `mirror.gcr.io/path` for example.

Note that you **can** specify a URL with scheme for this flag. Some valid options
are:

- `mirror.gcr.io`
- `127.0.0.1`
- `192.168.0.1:5000`
- `mycompany-docker-virtual.jfrog.io`
- `harbor.private.io/theproject`

#### Flag `--skip-default-registry-fallback`

Set this flag if you want the build process to fail if none of the mirrors
listed in flag [registry-mirror](#flag---registry-mirror) can pull some image.
This should be used with mirrors that implements a whitelist or some image
restrictions.

If [registry-mirror](#flag---registry-mirror) is not set or is empty, this flag
is ignored.

#### Flag `--reproducible`

Set this flag to strip timestamps out of the built image and make it
reproducible.

#### Flag `--secret`

Set this flag as `--secret id=MY_SECRET[,src=/file][,env=VAR][,type=file|env]` to configure build-secrets to be used during the build.

> [!IMPORTANT]
> The secret is **not stored securely** during the build and may be recoverable by other `RUN` steps even without explicitly mounting it. It should therefore not be considered confidential within the context of the build. The secret is never added to the image and never pushed.

#### Flag `--single-snapshot`

This flag takes a single snapshot of the filesystem at the end of the build, so
only one layer will be appended to the base image.

#### Flag `--skip-push-permission-check`

Set this flag to skip push permission check. This can be useful to delay Kanikos
first request for delayed network-policies.

#### Flag `--skip-tls-verify`

Set this flag to skip TLS certificate validation when pushing to a registry. It
is supposed to be used for testing purposes only and should not be used in
production!

#### Flag `--skip-tls-verify-pull`

Set this flag to skip TLS certificate validation when pulling from a registry.
It is supposed to be used for testing purposes only and should not be used in
production!

#### Flag `--skip-tls-verify-registry`

You can set `--skip-tls-verify-registry <registry-name>` to skip TLS certificate
validation when accessing the specified registry. It is supposed to be used for
testing purposes only and should not be used in production! You can set it
multiple times for multiple registries.

#### Flag `--snapshot-mode`

You can set the `--snapshot-mode=<full (default), redo, time>` flag to set how
kaniko will snapshot the filesystem.

- If `--snapshot-mode=full` is set, the full file contents and metadata are
  considered when snapshotting. This is the least performant option, but also
  the most robust.

- If `--snapshot-mode=redo` is set, the file mtime, size, mode, owner uid and
  gid will be considered when snapshotting. This may be up to 50% faster than
  "full", particularly if your project has a large number files.

- If `--snapshot-mode=time` is set, only file mtime will be considered when
  snapshotting (see [limitations related to mtime](#mtime-and-snapshotting)).

#### Flag `--tar-path`

Set this flag as `--tar-path=<path>` to save the image as a tarball at path. You
need to set `--destination` as well (for example `--destination=image`). If you
want to save the image as tarball only you also need to set `--no-push`.

A common use case is to scan the tarball for vulnerabilities before pushing.
The companion [`executor push`](#subcommand-push) subcommand pushes a tarball
produced by `--tar-path --no-push` without re-running the build:

```bash
# 1. Build to a tarball
/kaniko/executor --tar-path=/tmp/image.tar --no-push ...

# 2. Scan
trivy image --input /tmp/image.tar --exit-code 1

# 3. Push only if the scan passes — same binary, no extra tooling
/kaniko/executor push /tmp/image.tar --destination registry.example.com/myapp:latest
```

#### Flag `--target`

Set this flag to indicate which stages to build. If multiple targets are configured the first in the list is pushed.
If not set we implicitly target the last stage of the Dockerfile.

#### Flag `--use-new-run`

Using this flag enables an experimental implementation of the Run command which
does not rely on snapshotting at all. In this approach, in order to compute
which files were changed, a marker file is created before executing the Run
command. Then the entire filesystem is walked (takes ~1-3 seconds for 700Kfiles)
to find all files whose ModTime is greater than the marker file. With this new
run command implementation, the total build time is reduced seeing performance
improvements in the range of ~75%. This new run mode trades off
accuracy/correctness in some cases (potential for missed files in a "snapshot")
for improved performance by avoiding the full filesystem snapshots.

#### Flag `--verbosity`

#### Flag `--ignore-var-run`

Ignore /var/run when taking image snapshot. Set it to false to preserve
/var/run/\* in destination image. (Default true).

#### Flag `--ignore-path`

Set this flag as `--ignore-path=<path>` to ignore path when taking an image
snapshot. Set it multiple times for multiple ignore paths.

#### Flag `--image-fs-extract-retry`

Set this flag to the number of retries that should happen for the extracting an
image filesystem. Defaults to `0`.

#### Flag `--image-download-retry`

Set this flag to the number of retries that should happen when downloading the
remote image. Consecutive retries occur with exponential backoff and an initial
delay of 1 second. Defaults to `0`.

### Feature Flags

#### Profiles

##### Preview

Opting into the Preview profile gives you early access to upcoming performance improvements, bugfixes and features. While these flags are tested and ready to use, implementation details may still change.

```sh
FF_KANIKO_CROSS_REPO_MOUNT=true
FF_KANIKO_EXPAND_HEREDOC=true
FF_KANIKO_HASH_DIR_FRAMING=true
FF_KANIKO_INFER_CROSS_STAGE_CACHE_KEY=true
FF_KANIKO_PRECOMPILE_DOCKERIGNORE=true
FF_KANIKO_REPRODUCIBLE_PRESERVE_BASE_LAYERS=true
FF_KANIKO_RESOLVE_CACHE_KEY=true
FF_KANIKO_ROLLING_CACHE_KEY=true
FF_KANIKO_RUN_HONOR_GROUP=true
FF_KANIKO_RUN_VIA_TINI=true
FF_KANIKO_SHARED_BASE_CACHE=true
FF_KANIKO_SKIP_CACHED_STAGES=true
FF_KANIKO_SKIP_RELABEL_RECOMPRESS=true
FF_KANIKO_SKIP_WRITE_WHITEOUTS=true
FF_KANIKO_UNTAR_SKIP_ROOT=true
```

##### BuildKit compatibility

In a few places, Kaniko keeps its historical, non-compliant behaviour instead of following the Dockerfile specification. Switching to the compliant implementation by default would require users with large codebases to update their Dockerfiles, making migration to our fork tedious. These flags enable the spec-compliant behaviour, matching BuildKit one-to-one. Our integration tests run this profile on top of Preview.

```sh
FF_KANIKO_CHOWN_ON_IMPLICIT_DIRS=true
FF_KANIKO_COPY_AS_ROOT=true
FF_KANIKO_COPY_CHMOD_ON_IMPLICIT_DIRS=true
FF_KANIKO_EXPAND_HEREDOC=true
FF_KANIKO_RUN_HONOR_GROUP=true
FF_KANIKO_UNTAR_SKIP_ROOT=true
```

#### Flag `FF_KANIKO_COPY_AS_ROOT`

When files are copied from context, kaniko will copy them as the current user. But according to [dockerfile specification](https://docs.docker.com/reference/dockerfile/#copy---chown---chmod) they should always be copied as `root:root` unless specified otherwise.
Set this flag to `true` to implement COPY as specified. Defaults to `false`.
Currently no plans to activate.

#### Flag `FF_KANIKO_IGNORE_CACHED_MANIFEST`

Warmer does not only store the image as a tarball, but also the original manifest as a separate json file.
This is done to speedup manifest retrieval, but has adverse effects in some scenarios, as storing the image as a tarball actively rewrites part of the image, specifically it forces the mediatype to `vnd.docker.distribution.manifest.v2+json`. This causes the stored manifest being incompatible with the stored image. With this featureflag we ignore the manifest stored in cache and instead create the manifest from the image upon load.
Set this flag to `true` to ignore stored manifest.json in the cache directory. Defaults to `false`.
Currently no plans to activate.

#### Flag `FF_KANIKO_RUN_MOUNT_BIND`

Set this flag to `true` to enable bind mounts in `RUN` statements, ie.
```dockerfile
RUN --mount=type=bind,source=requirements.txt,target=/tmp/requirements.txt \
 uv pip install -r /tmp/requirements.txt
```
cross-stage bind mounts `from=<stage>` are not yet supported.
Defaults to `true`.
Will be deprecated in `v1.29.0`.

#### Flag `FF_KANIKO_DISABLE_HTTP2`

We noticed that there is a significant performance gap when using http/2.0 together with gitlab registry. Set this flag to `true` to enforce http/1.1 protocol, the same behaviour as if setting `GODEBUG="http2client=0"`.
Defaults to `false`.
Currently no plans to activate.

#### Flag `FF_KANIKO_OCI_WARMER`

Warmer stores images in a tarball via go-containerregistry. However, this approach creates two problems. The tarball writer only supports dockerv2 mediatype, so building from warmer cache might result in a different output image than building from remote, as we forcefully rewrite all images to that mediatype. Secondly, the performance/usability of that approach is suboptimal, as we either store the manifest in a separate file, causing consistency issues or recalculate upon load (see [`FF_KANIKO_IGNORE_CACHED_MANIFEST`](#flag-ff_kaniko_ignore_cached_manifest)). With this change we use ocilayout instead. Ocilayout folders support arbitrary mediatypes and store the manifest alongside the image data.
Set this flag to `true` to store warmer cache images as ocilayout. Note that this flag has to be passed to both warmer and executor.
Defaults to `true`.
Will be deprecated in `v1.29.0`.

#### Flag `FF_KANIKO_RUN_VIA_TINI`

Kaniko usually runs as PID1 in the container, but kaniko currently does not implement reaping of zombie processes, nor does it offload that task to the kernel. As a result, any short-lived child processes spawned by your `RUN` command may linger around as zombies and potentially cause your build to hang.
Set this flag to `true` to run any `RUN` commands via `tini` init system as subreaper, to properly handle zombie processes.
Note that for this feature to work the tini binary must be available as `/kaniko/tini`.
Defaults to `false`.
Becomes default in `v1.29.0`, once the `tini` binary ships for RISC-V.

#### Flag `FF_KANIKO_COPY_CHMOD_ON_IMPLICIT_DIRS`

When files are copied into a non-existing directory, both kaniko and buildkit will create the directory and all required parent directories implicitly. If chmod option is given, buildkit will apply the chmod not only on the copied files & folders, but on all implicit parent dirs too. Kaniko will use regular folder permissions (0755) on parent directories instead and only apply the chmod on the explicitly created files & folders.
Set this flag to `true` to implement COPY chmod like buildkit. Defaults to `false`.
Currently no plans to activate.

#### Flag `FF_KANIKO_CHOWN_ON_IMPLICIT_DIRS`

When `WORKDIR` creates a directory whose parents do not exist yet, kaniko creates the parents but leaves them owned by root, assigning the active `USER` only to the final directory. buildkit assigns the active `USER` to every directory it creates, so `WORKDIR /work/dir` under `USER 1000` leaves `/work` owned by root in kaniko but by `1000` in buildkit.
Set this flag to `true` to chown every implicitly created directory to the active user, matching buildkit. Defaults to `false`.
Currently no plans to activate.

#### Flag `FF_KANIKO_CLEAN_KANIKO_DIR`

When using `--cleanup`, kaniko cleans the container filesystem at the end of the build. Set this flag to `true` to also remove kaniko's own working directory artifacts from `/kaniko` (the Dockerfile copy, build context, intermediate stages, inter-stage dependencies, layers cache, and secrets). This is useful when reusing a kaniko container across multiple builds.
Defaults to `true`.

#### Flag `FF_KANIKO_NO_PROPAGATE_ANNOTATIONS`

When building from a base image that carries OCI manifest annotations (e.g. `org.opencontainers.image.url`, `org.opencontainers.image.version`), kaniko by default propagates those annotations into the output image manifest. This differs from Docker/BuildKit behaviour, which does not carry base image annotations forward into derived images.
Set this flag to `true` to strip base image manifest annotations from the output, matching Docker behaviour. Defaults to `true`.
Will be deprecated in `v1.29.0`.

#### Flag `FF_KANIKO_OCI_SCRATCH_BASE`

When a Dockerfile uses `FROM scratch`, kaniko uses an empty Docker-format image as the build base, which means the output image is produced in Docker manifest schema v2 format.
Set this flag to `true` to use an empty OCI-format image instead, causing `FROM scratch` builds to produce output in OCI manifest schema v1 format. Defaults to `false`.
Currently no plans to activate.

#### Flag `FF_KANIKO_VOLUME_SKIP_MKDIR`

Kaniko creates the directory declared by `VOLUME` on the filesystem; Docker/BuildKit does not.
This causes a cache bug in multistage builds, the directory gets a fresh `mtime` on every run, which breaks cache hits in downstream stages.
Set this flag to `true` to skip the implicit directory creation, matching Docker/BuildKit behaviour. Defaults to `true`.
Will be deprecated in `v1.29.0`.

#### Flag `FF_KANIKO_PRESERVE_HARDLINKS`

When copying a directory via `COPY --from=<stage>`, kaniko copies each file independently, breaking hardlink relationships. Files that shared a single inode in the source stage become independent copies in the output image, which can significantly inflate image size for images that rely heavily on hardlinks (e.g. `git` installations where many binaries are hardlinked together).
Set this flag to `true` to preserve hardlinks during `COPY --from`. Defaults to `true`.
Will be deprecated in `v1.29.0`.

#### Flag `FF_KANIKO_RELATIVE_LINK_TARGETS`

When a snapshot layer contains a hardlink, kaniko writes the link target as an absolute path while writing the entry name itself relative to the tar root. Docker writes both relative. Older clients accept either form and extract them to the same file, but docker `29.7.0` rejects the absolute one with `invalid hardlink target "/usr/bin/unzip"`, so affected images can no longer be loaded or pulled.
Set this flag to `true` to write hardlink targets relative to the tar root.
Defaults to `true`.
Will be deprecated in `v1.29.0`.

#### Flag `FF_KANIKO_SKIP_WRITE_WHITEOUTS`

When kaniko extracts a cached layer it applies the layer's whiteouts by deleting the target files, but it also writes the `.wh.<name>` marker files onto the working filesystem. With `--cache-copy-layers` a later cross-stage `COPY --from=<stage>` copies such a marker verbatim and commits it as a real whiteout, so a cache-hit build deletes a file that the cache-miss build kept.
Set this flag to `true` to skip writing the marker files, the deletion alone already applies the whiteout. Defaults to `false`.
Becomes default in `v1.29.0`.

#### Flag `FF_KANIKO_BUILDKIT_ARG_ENV_PRECEDENCE`

The [Dockerfile spec](https://docs.docker.com/reference/dockerfile/#using-arg-variables) states that an `ENV` instruction overrides an `ARG` of the same name. This is correct but order-dependent: "override" implies there is already a value to override, so the rule applies when `ENV` appears *after* `ARG`. Applied consistently, an `ARG` declared after an `ENV` (including one inherited from a base image) should win, which is the behaviour BuildKit implements.

Kaniko's legacy behaviour treats `ENV` as unconditionally winning regardless of declaration order. Enable this flag to match BuildKit semantics, where the later declaration takes precedence:

```dockerfile
FROM alpine AS base
ENV HELLO=upstream

FROM base AS child
ARG HELLO
RUN echo $HELLO   # prints the --build-arg value, not "upstream"
```

Set this flag to `true` to enable BuildKit-compatible ARG/ENV precedence. Defaults to `true`.
Will be deprecated in `v1.29.0`.

#### Flag `FF_KANIKO_INFER_CROSS_STAGE_CACHE_KEY`

When a multi-stage build uses `COPY --from=<stage>`, kaniko normally hashes the copied files from the source stage's filesystem to compute the downstream cache key. The source stage's `finalCacheKey` is a deterministic function of its build inputs and can be used as a stable proxy for those file contents, so the downstream cache key can be inferred without accessing the filesystem at all. This is a preparatory optimisation for a future change that will avoid unpacking the source stage's filesystem entirely when all downstream stages are also fully cached.
Set this flag to `true` to add additional cache entries for the shortcuts, currently they do not yet allow optimization.
Requires `--cache-copy-layers`. Defaults to `false`.
Becomes default in `v1.29.0`.

#### Flag `FF_KANIKO_CACHE_LOOKAHEAD`

Set this flag to `true` to run a precompute pass before the build loop that derives each stage's final cache key ahead of time. The build loop still recomputes each key during its own `optimize()` call and asserts that it matches the precomputed value. This is a developer assertion to verify the new precompute pass is correct, there is no benefit to enabling it in production.
Defaults to `false`.

#### Flag `FF_KANIKO_ROLLING_CACHE_KEY`

By default the composite cache joins all inputs with `-` and hashes the result. Since `-` is a legal input too, different sequences can join to the same text and collide, silently picking up the wrong cache layer.
This flag switches to a rolling hash, think git, where `state = SHA256(state || input)`. This makes boundaries unambiguous and prevents collisions. It also prevents the hash input from growing, so key computation gets marginally cheaper. Toggling it changes every cache key, forcing a rebuild from scratch.
Set this flag to `true` to enable the rolling cache key.
Defaults to `false`.
Becomes default in `v1.29.0`.

#### Flag `FF_KANIKO_HASH_DIR_FRAMING`

Directory cache keys concatenate each relative path and file hash without recording their boundaries. A filename can therefore absorb an adjacent hash and make distinct directory trees produce the same cache key, silently reusing the wrong cached layer.
Set this flag to `true` to length-prefix every path and file hash before hashing the directory. Toggling it changes cache keys for `COPY` and `ADD` directory inputs, forcing those layers to rebuild once.
Defaults to `false`.
Becomes default in `v1.29.0`.

#### Flag `FF_KANIKO_CACHE_PROBE_AFTER_MISS`

By default, a single layer cache miss within a stage disables every subsequent cache lookup in that same stage. A 30-step Dockerfile with one transient miss at step 3 will rebuild the remaining 27 layers locally even when later layers are still cached in the registry.

Set this flag to `true` to keep probing the cache after a miss. Cached tar diffs apply cleanly on top of locally-rebuilt prior layers under the same determinism assumption the cache scheme already requires (cache keys encode commands and inputs, not prior-layer outputs).

The other `stopCache` site — `COPY --from` in the precompute pass — is intentionally not affected by this flag. That one signals "key cannot be computed without the file context", not a transient miss, and is required for correctness.
Defaults to `false`.

#### Flag `FF_KANIKO_WARMER_CACHE_LOCK`

Multiple warmer processes sharing a cache volume can race when warming the same image; with [`FF_KANIKO_OCI_WARMER`](#flag-ff_kaniko_oci_warmer) one of them may exit with an error.
Set this flag to `true` to coordinate concurrent warmers and avoid redundant downloads. Corrupt or wrong-format cache entries are detected and replaced, making [`FF_KANIKO_OCI_WARMER`](#flag-ff_kaniko_oci_warmer) toggles transparent.
Defaults to `true`.
Will be deprecated in `v1.29.0`.

#### Flag `FF_KANIKO_PRESERVE_MOUNTED_PATHS`

When a container runtime bind-mounts files read-only into the build container — as the NVIDIA GPU operator does with driver artifacts (`nvidia-smi`, `libnvidia*`, firmware blobs) on GPU nodes — and a base image layer ships a directory along that mount path as a symlink, kaniko `os.RemoveAll`s the directory while unpacking to make way for the symlink. The recursive remove hits the read-only bind mount and the build fails with `unlinkat ...: device or resource busy`.
Set this flag to `true` to skip removing a directory that contains a mounted (ignored) path: its other contents are still cleared, but the mount is preserved and the conflicting layer entry is left in place, matching how `DeleteFilesystem` already treats mounts. Defaults to `true`.
Will be deprecated in `v1.29.0`.

#### Flag `FF_KANIKO_REPRODUCIBLE_PRESERVE_BASE_LAYERS`

`--reproducible` re-tars every layer to zero its timestamps, including layers inherited from the `FROM` image. Base-layer blobs get fresh digests on every build and stop matching the upstream registry, defeating layer reuse even though kaniko changed nothing in them.
Set this flag to `true` to re-time only kaniko-appended layers and pass base layers through unchanged.
Defaults to `false`.
Becomes default in `v1.29.0`.

#### Flag `FF_KANIKO_DEPRECATE_INTER_STAGE_RESTORE`

Deprecates the inter-stage restore performed by [`--preserve-context`](#flag---preserve-context) when used without [`--pre-cleanup`](#flag---pre-cleanup). Set to `1` to fully disable the restore between stages. The original motivation, smuggling secrets across stages, is now served by `RUN --mount=type=secret`.
Defaults to `true`.
Will be deprecated in `v1.29.0`.

#### Flag `FF_KANIKO_SCOPED_DOCKERIGNORE`

Scopes `.dockerignore` patterns to the build context. Without it, an absolute path outside the context (such as an intermediate `/kaniko/...` source of a multi-stage `COPY --from`) is matched against the patterns; with an allowlist `.dockerignore` (e.g. `*` then `!keep`) the catch-all wrongly excludes it, dropping it from the layer cache key and serving a stale image when only that file changed. Set this flag to `true` to never match paths outside the context.
Defaults to `false`.
Currently no plans to activate.

#### Flag `FF_KANIKO_PRECOMPILE_DOCKERIGNORE`

`.dockerignore` rebuilds the pattern matcher and recompiles every glob to a regexp on every file. Set this flag to `true` to pre-compile the matcher instead.
Defaults to `false`.
Becomes default in `v1.29.0`.

#### Flag `FF_KANIKO_RESOLVE_CACHE_KEY`

A `COPY`, `ADD` or `WORKDIR` layer cache key is built from the raw instruction text, so build args and env expanded in the instruction (for example `COPY foo /$A/foo` or `WORKDIR /$A/foo`) do not enter the key. A build that only changes such a variable can hit stale cache entries.
Set this flag to `true` to interpolate build args and env.
Defaults to `false`.
Becomes default in `v1.29.0`.

#### Flag `FF_KANIKO_SKIP_RELABEL_RECOMPRESS`

When a cached layer is reused in an image of a different media-type vendor, kaniko not only relabels the layer but re-gzips it too. if the compression is unchanged, the re-encoded blob is byte-identical to the original.
Set this flag to `true` to skip the unecessary re-compression and serve the relabeled already-compressed blob.
Defaults to `false`.
Becomes default in `v1.29.0`.

#### Flag `FF_KANIKO_SECUREJOIN_EXTRACTION`

When unpacking image layers kaniko joins each tar entry path lexically, so a malicious base image can ship an escaping symlink followed by a write through it and land the write outside the extraction root, for example overwriting `/kaniko/tini` to gain RCE.
Set this flag to `true` to resolve each entry's parent with SecureJoin so the write stays contained inside the destination.
Defaults to `true`.
Will be deprecated in `v1.29.0`.

#### Flag `FF_KANIKO_UNTAR_SKIP_ROOT`

When `ADD` extracts a local tar archive into a directory, kaniko applies the archive's root `.` entry to the destination directory and overwrites its mode and ownership, while docker leaves the destination untouched.
Set this flag to `true` to skip the root `.` entry when untarring.
Defaults to `false`.
Becomes default in `v1.29.0`.

#### Flag `FF_KANIKO_RUN_HONOR_GROUP`

When a stage sets `USER user:group`, `RUN` applies only the user and the gid falls back to the user's primary group, so an explicit group is silently dropped.
Set this flag to `true` to pass the full `user:group` to the command so `RUN` runs with the requested group, matching docker.
Defaults to `false`.
Becomes default in `v1.29.0`.

#### Flag `FF_KANIKO_EXPAND_HEREDOC`

Docker applies Dockerfile word-expansion to a `COPY` or `ADD` heredoc body when the delimiter is unquoted, so `${VAR}` expands and `\${VAR}` keeps the literal text. A quoted delimiter (`<<'EOF'`) leaves the body verbatim. kaniko writes the body verbatim in every case, so the expanded files diverge from Docker.
Set this flag to `true` to expand build args and env in unquoted `COPY` and `ADD` heredoc bodies.
Defaults to `false`.
Becomes default in `v1.29.0`.

#### Flag `FF_KANIKO_SKIP_CACHED_STAGES`

When a multi-stage build uses `COPY --from=<stage>`, the downstream cache key depends on the copied files. So the entire source stage has to be built and unpacked, only to then realize that we had a cache hit and can throw away the upstream stage. We recently introduced `FF_KANIKO_INFER_CROSS_STAGE_CACHE_KEY`, `FF_KANIKO_CACHE_LOOKAHEAD` and `FF_KANIKO_ROLLING_CACHE_KEY`, with that we can know a-priori whether we will have a cache hit or not. `FF_KANIKO_SKIP_CACHED_STAGES` is the logical conclusion then, it simply runs another elision pass over the now updated list of stages and drops all stages that are no longer required to be built. Where a key cannot be inferred the stage is built as before. A fully cached build collapses into a single stage with nothing to unpack.
Set this flag to `true` to run the second elision pass.
Defaults to `false`.
Becomes default in `v1.29.0`.

#### Flag `FF_KANIKO_SHARED_BASE_CACHE`

When several stages build on the same remote base image, kaniko downloads that base once per stage. Set this flag to `true` to download a shared base once, store it under `/kaniko/bases`, and have the other stages read it from there instead of downloading it again. A base is also stored when a stage is kept for a later stage to build on, or when the built image is pushed, because both re-read the base layers. A base used by a single stage that is not pushed still streams, so nothing is stored that would not be read again.
Stored bases stay in `/kaniko/bases` after the build, `--cleanup` does not remove them. A long-lived executor that runs many builds therefore accumulates every base it ever downloaded.
Defaults to `false`.
Becomes default in `v1.29.0`.

#### Flag `FF_KANIKO_CROSS_REPO_MOUNT`

A registry can copy a blob between its own repositories for free, but only if it is told which repository already holds it. Kaniko loses that as soon as it copies a layer locally. Worse, with `--cache` every built layer goes up twice, once to the cache repo and once inside the image. Set this flag to `true` to remember which layers can be mounted remotely and which ones genuinely need to be pushed.
Defaults to `false`.
Becomes default in `v1.29.0`.

### Assertion Overrides

Kaniko checks internal invariants at runtime. If one is violated the build stops with a message like:

```
Assertion violated [executor.build.metadata-only]: ...
```

This is always a bug in kaniko, please [open an issue](https://github.com/osscontainertools/kaniko/issues).

As a temporary workaround, pass the name in brackets to `KANIKO_IGNORE_ASSERTIONS` to skip that assertion and log a warning instead:

```sh
KANIKO_IGNORE_ASSERTIONS=executor.build.metadata-only
```

Multiple names can be passed as a comma-separated list.

### Telemetry

Kaniko can export an OpenTelemetry trace of each build so you can track build times, cache hits and misses, and which Dockerfile instructions keep busting the cache. It is off by default and a no-op unless you point it at a collector:

```sh
KANIKO_TELEMETRY_ENDPOINT=http://otel-collector:4318
```

Each build becomes a trace, with a span per build phase and Dockerfile command. Telemetry is best effort and never fails a build.

**What leaves the machine**: every trace carries the full Dockerfile source (`kaniko.dockerfile.content`), the verbatim text of every instruction, the values of any explicitly-set `FF_KANIKO_*` flags, and cache keys, all unredacted. Nothing beyond that is captured: the runtime value behind a `RUN --mount=type=secret` or the contents of a `--mount=type=cache` never reach a trace. In case your Dockerfile and `RUN` themselves contain credentials, treat the collector as part of your secret boundary.

Spans are sent over OTLP/**HTTP(S)**, OTLP/**gRPC** is not supported. The endpoint URL must include a scheme, and only `KANIKO_TELEMETRY_ENDPOINT` enables tracing, the standard `OTEL_EXPORTER_OTLP_ENDPOINT` alone does not.

See [Telemetry attributes](docs/telemetry.md) for the full list of exported span attributes.

Standard OpenTelemetry environment variables apply for the rest: `OTEL_EXPORTER_OTLP_HEADERS` for authenticating to the collector, and `OTEL_RESOURCE_ATTRIBUTES` for fleet labels such as `tenant`, `repo`, and `git.sha`.

### Debug Image

The kaniko executor image is based on scratch and doesn't contain a shell. We
provide `gcr.io/kaniko-project/executor:debug`, a debug image which consists of
the kaniko executor image along with a busybox shell to enter.

You can launch the debug image with a shell entrypoint:

```shell
docker run -it --entrypoint=/busybox/sh ghcr.io/osscontainertools/kaniko:debug
```

## Security

kaniko by itself **does not** make it safe to run untrusted builds inside your
cluster, or anywhere else.

kaniko relies on the security features of your container runtime to provide
build security.

The minimum permissions kaniko needs inside your container are governed by a few
things:

- The permissions required to unpack your base image into its container
- The permissions required to execute the RUN commands inside the container

If you have a minimal base image (SCRATCH or similar) that doesn't require
permissions to unpack, and your Dockerfile doesn't execute any commands as the
root user, you can run kaniko without root permissions. It should be noted that
Docker runs as root by default, so you still require (in a sense) privileges to
use kaniko.

You may be able to achieve the same default seccomp profile that Docker uses in
your Pod by setting
[seccomp](https://kubernetes.io/docs/concepts/policy/pod-security-policy/#seccomp)
profiles with annotations on a
[PodSecurityPolicy](https://cloud.google.com/kubernetes-engine/docs/how-to/pod-security-policies)
to create or update security policies on your cluster.

### Verifying Signed Kaniko Images

kaniko images are signed for versions >= 1.24.1 using
[cosign](https://github.com/sigstore/cosign)!

To verify a public image, install [cosign](https://github.com/sigstore/cosign)
and use keyless verification with github actions issuer.
Note that to verify images after v1.26.0 you require cosign v3.

```
$ cosign verify \
  --certificate-identity-regexp "https://github.com/osscontainertools/kaniko/.github/workflows/images.yaml@.*" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  ghcr.io/osscontainertools/kaniko:latest
```

kaniko images for older versions [1.24.1 - 1.25.5] can be verified using
```
$ cosign verify \
  --certificate-identity-regexp "https://github.com/mzihlmann/kaniko/.github/workflows/images.yaml@.*" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  ghcr.io/osscontainertools/kaniko:latest
```

## Creating Multi-arch Container Manifests Using Kaniko and Manifest-tool

While Kaniko itself currently does not support creating multi-arch manifests
(contributions welcome), one can use tools such as
[manifest-tool](https://github.com/estesp/manifest-tool) to stitch multiple
separate builds together into a single container manifest.

### General Workflow

The general workflow for creating multi-arch manifests is as follows:

1. Build separate container images using Kaniko on build hosts matching your
   target architecture and tag them with the appropriate ARCH tag.
2. Push the separate images to your container registry.
3. Manifest-tool identifies the separate manifests in your container registry,
   according to a given template.
4. Manifest-tool pushes a combined manifest referencing the separate manifests.

![Workflow Multi-arch](https://raw.githubusercontent.com/osscontainertools/kaniko/main/docs/images/multi-arch.drawio.svg)

### Limitations and Pitfalls

The following conditions must be met:

1. You need access to build-machines running the desired architectures (running
   Kaniko in an emulator, e.g. QEMU should also be possible but goes beyond the
   scope of this documentation). This is something to keep in mind when using
   SaaS build tools such as github.com or gitlab.com, of which at the time of
   writing neither supports any non-x86_64 SaaS runners
   ([GitHub](https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources),[GitLab](https://docs.gitlab.com/ee/ci/runners/saas/linux_saas_runner.html#machine-types-available-for-private-projects-x86-64)),
   so be prepared to bring your own machines
   ([GitHub](https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners/about-self-hosted-runners),[GitLab](https://docs.gitlab.com/runner/register/).
2. Kaniko needs to be able to run on the desired architectures. At the time of
   writing, the official Kaniko container supports
   [linux/amd64, linux/arm64, linux/s390x and linux/ppc64le (not on \*-debug images)](https://github.com/osscontainertools/kaniko/blob/main/.github/workflows/images.yaml).
3. The container registry of your choice must be OCIv1 or Docker v2.2
   compatible.

### Example CI Pipeline (GitLab)

It is up to you to find an automation tool that suits your needs best. We
recommend using a modern CI/CD system such as GitHub workflows or GitLab CI. As
we (the authors) happen to use GitLab CI, the following examples are tailored to
this specific platform but the underlying principles should apply anywhere else
and the examples are kept simple enough, so that you should be able to follow
along, even without any previous experiences with this specific platform. When
in doubt, visit the
[gitlab-ci.yml reference page](https://docs.gitlab.com/ee/ci/yaml/index.html)
for a comprehensive overview of the GitLab CI keywords.

#### Building the Separate Container Images

gitlab-ci.yml:

```yaml
# define a job for building the containers
build-container:
  stage: container-build
  # run parallel builds for the desired architectures
  parallel:
    matrix:
      - ARCH: amd64
      - ARCH: arm64
  tags:
    # run each build on a suitable, preconfigured runner (must match the target architecture)
    - runner-${ARCH}
  image:
    name: ghcr.io/osscontainertools/kaniko:debug
    entrypoint: [""]
  script:
    # build the container image for the current arch using kaniko
    - >-
      /kaniko/executor --context "${CI_PROJECT_DIR}" --dockerfile
      "${CI_PROJECT_DIR}/Dockerfile" # push the image to the GitLab container
      registry, add the current arch as tag. --destination
      "${CI_REGISTRY_IMAGE}:${ARCH}"
```

#### Merging the Container Manifests

gitlab-ci.yml:

```yaml
# define a job for creating and pushing a merged manifest
merge-manifests:
  stage: container-build
  # all containers must be build before merging them
  # alternatively the job may be configured to run in a later stage
  needs:
    - job: container-build
      artifacts: false
  tags:
    # may run on any architecture supported by manifest-tool image
    - runner-xyz
  image:
    name: mplatform/manifest-tool:alpine
    entrypoint: [""]
  script:
    - >-
      manifest-tool # authorize against your container registry
      --username=${CI_REGISTRY_USER} --password=${CI_REGISTRY_PASSWORD} push
      from-args # define the architectures you want to merge --platforms
      linux/amd64,linux/arm64 # "ARCH" will be automatically replaced by
      manifest-tool # with the appropriate arch from the platform definitions
      --template ${CI_REGISTRY_IMAGE}:ARCH # The name of the final, combined
      image which will be pushed to your registry --target ${CI_REGISTRY_IMAGE}
```

#### On the Note of Adding Versioned Tags

For simplicity's sake we deliberately refrained from using versioned tagged
images (all builds will be tagged as "latest") in the previous examples, as we
feel like this adds to much platform and workflow specific code.

Nevertheless, for anyone interested in how we handle (dynamic) versioning in
GitLab, here is a short rundown:

- If you are only interested in building tagged releases, you can simply use the
  [GitLab predefined](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)
  `CI_COMMIT_TAG` variable when running a tag pipeline.
- When you (like us) want to additionally build container images outside of
  releases, things get a bit messier. In our case, we added a additional job
  which runs before the build and merge jobs (don't forget to extend the `needs`
  section of the build and merge jobs accordingly), which will set the tag to
  `latest` when running on the default branch, to the commit hash when run on
  other branches and to the release tag when run on a tag pipeline.

gitlab-ci.yml:

```yaml
container-get-tag:
  stage: pre-container-build-stage
  tags:
    - runner-xyz
  image: busybox
  script:
    # All other branches are tagged with the currently built commit SHA hash
    - |
      # If pipeline runs on the default branch: Set tag to "latest"
      if test "$CI_COMMIT_BRANCH" == "$CI_DEFAULT_BRANCH"; then
        tag="latest"
      # If pipeline is a tag pipeline, set tag to the git commit tag
      elif test -n "$CI_COMMIT_TAG"; then
        tag="$CI_COMMIT_TAG"
      # Else set the tag to the git commit sha
      else
        tag="$CI_COMMIT_SHA"
      fi
    - echo "tag=$tag" > build.env
  # parse tag to the build and merge jobs.
  # See: https://docs.gitlab.com/ee/ci/variables/#pass-an-environment-variable-to-another-job
  artifacts:
    reports:
      dotenv: build.env
```

## Comparison with Other Tools

Similar tools include:

- [BuildKit](https://github.com/moby/buildkit)
- [img](https://github.com/genuinetools/img)
- [orca-build](https://github.com/cyphar/orca-build)
- [umoci](https://github.com/openSUSE/umoci)
- [buildah](https://github.com/containers/buildah)
- [FTL](https://github.com/GoogleCloudPlatform/runtimes-common/tree/master/ftl)
- [Bazel rules_docker](https://github.com/bazelbuild/rules_docker)

All of these tools build container images with different approaches.

BuildKit (and `img`) can perform as a non-root user from within a container but
requires seccomp and AppArmor to be disabled to create nested containers.
`kaniko` does not actually create nested containers, so it does not require
seccomp and AppArmor to be disabled. BuildKit supports "cross-building"
multi-arch containers by leveraging QEMU.

`orca-build` depends on `runc` to build images from Dockerfiles, which can not
run inside a container (for similar reasons to `img` above). `kaniko` doesn't
use `runc` so it doesn't require the use of kernel namespacing techniques.
However, `orca-build` does not require Docker or any privileged daemon (so
builds can be done entirely without privilege).

`umoci` works without any privileges, and also has no restrictions on the root
filesystem being extracted (though it requires additional handling if your
filesystem is sufficiently complicated). However, it has no `Dockerfile`-like
build tooling (it's a slightly lower-level tool that can be used to build such
builders -- such as `orca-build`).

`Buildah` specializes in building OCI images. Buildah's commands replicate all
of the commands that are found in a Dockerfile. This allows building images with
and without Dockerfiles while not requiring any root privileges. Buildah’s
ultimate goal is to provide a lower-level coreutils interface to build images.
The flexibility of building images without Dockerfiles allows for the
integration of other scripting languages into the build process. Buildah follows
a simple fork-exec model and does not run as a daemon but it is based on a
comprehensive API in golang, which can be vendored into other tools.

`FTL` and `Bazel` aim to achieve the fastest possible creation of Docker images
for a subset of images. These can be thought of as a special-case "fast path"
that can be used in conjunction with the support for general Dockerfiles kaniko
provides.

## Limitations

### mtime and snapshotting

When taking a snapshot, kaniko's hashing algorithms include (or in the case of
[`--snapshot-mode=time`](#--snapshotmode), only use) a file's
[`mtime`](https://en.wikipedia.org/wiki/Inode#POSIX_inode_description) to
determine if the file has changed. Unfortunately, there is a delay between when
changes to a file are made and when the `mtime` is updated. This means:

- With the time-only snapshot mode (`--snapshot-mode=time`), kaniko may miss
  changes introduced by `RUN` commands entirely.
- With the default snapshot mode (`--snapshot-mode=full`), whether or not kaniko
  will add a layer in the case where a `RUN` command modifies a file **but the
  contents do not** change is theoretically non-deterministic. This _does not
  affect the contents_ which will still be correct, but it does affect the
  number of layers.

_Note that these issues are currently theoretical only. If you see this issue
occur, please
[open an issue](https://github.com/osscontainertools/kaniko/issues)._

### Dockerfile commands `--chown` support
Kaniko currently supports `COPY --chown` and `ADD --chown` Dockerfile command. It does not support `RUN --chown`.

## References

- [Kaniko - Building Container Images In Kubernetes Without Docker](https://youtu.be/EgwVQN6GNJg).

kaniko

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README