KO | EN

GitLite

Minimalist Open Source Repository Search Engine

gitlite — search

#dsh #dsh-plugin #ai-agents #developer-tools #codex #claude-code #mcp #rust #deepseek-harness #typescript #ai-agent #cli

gke-metadata-server

★ 43 Open GitHub ↗

A Kubernetes DaemonSet for emulating GKE Workload Identity in non-GKE clusters.

// repository documentation

Was this content helpful?

★ 0 (0 ratings)

## GA is here!

Please [check it out](https://github.com/matheuscscp/gke-metadata-server/discussions/460)!

# gke-metadata-server

![release](https://img.shields.io/github/v/release/matheuscscp/gke-metadata-server?style=flat&color=blue)
[![SLSA 3](https://slsa.dev/images/gh-badge-level3.svg)](https://slsa.dev)
![FIPS 140-3 compliant](https://img.shields.io/badge/FIPS_140--3-compliant-brightgreen)
![Distroless](https://img.shields.io/badge/Distroless-base_image-blue)

A GKE Metadata Server *emulator* for making it easier to use GCP Workload Identity Federation
inside *non-GKE* Kubernetes clusters, e.g. KinD, on-prem, managed Kubernetes from other
clouds, or even self-managed clusters deployed in GCE VMs. This implementation tries to mimic the
`gke-metadata-server` `DaemonSet` deployed automatically by Google in the `kube-system` namespace
of GKE clusters that have the feature *Workload Identity Federation for GKE* enabled. See how the
GKE Metadata Server
[works](https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity#metadata_server).

## Usage

Steps:
1. Configure GCP Workload Identity Federation for Kubernetes.
2. Deploy `gke-metadata-server` in the cluster using the Workload Identity Provider full name,
obtained after step 1.
3. Add the label `iam.gke.io/gke-metadata-server-enabled: "true"` to the Nodes you want to have
the emulator Pods running on. Only Pods running on these Nodes will be able to use the emulator.
The most recommended way to do this is by using Node pooling/grouping features from your Kubernetes
provider. Alternatively, set `requireNodeLabel: false` to skip this label requirement and run the
emulator on all nodes.
4. See [`./testdata/pod.yaml`](./testdata/pod.yaml) for an example of how to configure your Pods
and their ServiceAccounts.
5. (Optional but highly recommended) Verify gke-metadata-server's artifact signatures to make sure
you are deploying authentic artifacts distributed by this project.

### Configure GCP Workload Identity Federation for Kubernetes

Steps:
1. Create a Workload Identity Pool and Provider pair for the cluster.
2. Grant Kubernetes ServiceAccounts permission to impersonate Google Service Accounts.

Official docs and examples are
[here](https://cloud.google.com/iam/docs/workload-identity-federation-with-kubernetes#kubernetes).

More examples for all the configuration described in this section are available here:
[`./terraform/test.tf`](./terraform/test.tf). This is where we provision the
infrastructure required for testing this project in CI and development.

#### Create a Workload Identity Pool and Provider pair for the cluster

Before granting ServiceAccounts of a Kubernetes cluster permission to impersonate Google
Service Accounts, a Workload Identity Pool and Provider pair must be created for the cluster
in Google Cloud Platform.

A Pool is a namespace for Subjects and Providers, where Subjects represent the identities to
whom permission to impersonate Google Service Accounts are granted, i.e. the Kubernetes
ServiceAccounts in this case.

A Provider stores the OpenID Connect configuration parameters retrieved from the cluster for
allowing Google to verify ServiceAccount Tokens issued by the cluster. Specifically for the
creation of a Provider, see an example in the [`./Makefile`](./Makefile), target
`create-or-update-provider`.

The ServiceAccount Tokens issued by Kubernetes are JWT tokens. Each such token is exchanged
for an OAuth 2.0 Access Token that impersonates a Google Service Account. During a token
exchange, the Subject is mapped from the `sub` claim of the JWT (achieved in the `Makefile`
example by `--attribute-mapping=google.subject=assertion.sub`). In this case the Subject
has the following format:

`system:serviceaccount:{k8s_namespace}:{k8s_sa_name}`

The Subject total length cannot exceed 127 characters
([docs](https://cloud.google.com/iam/docs/reference/rest/v1/projects.locations.workloadIdentityPools.providers#WorkloadIdentityPoolProvider.FIELDS.attribute_mapping)).

**Attention 1:** If you update the private keys of the ServiceAccount Issuer of the cluster
you must update the Provider with the new OpenID Connect configuration, *otherwise service
will be disrupted*.

**Attention 2:** Please make sure not to specify any audiences when creating the Provider.
The emulator uses the *default audience* when issuing the ServiceAccount Tokens. The default
audience contains the full name of the Provider, which is a strong restriction:

`//iam.googleapis.com/{provider_full_name}`

where `{provider_full_name}` has the form:

`{pool_full_name}/providers/{provider_short_name}`

and `{pool_full_name}` has the form:

`projects/{gcp_project_number}/locations/global/workloadIdentityPools/{pool_short_name}`

#### Grant Kubernetes ServiceAccounts permission to impersonate Google Service Accounts

For allowing the Kubernetes ServiceAccount `{k8s_sa_name}` from the namespace `{k8s_namespace}`
to impersonate the Google Service Account `{gcp_service_account}@{gcp_project_id}.iam.gserviceaccount.com`,
grant the IAM Role `roles/iam.workloadIdentityUser` on this Google Service Account to the following
principal:

`principal://iam.googleapis.com/{pool_full_name}/subject/system:serviceaccount:{k8s_namespace}:{k8s_sa_name}`

This principal will be reflected as a Subject in the Google Cloud Console webpage of the Pool.

#### Alternatively, grant direct resource access to the Kubernetes ServiceAccount

Workload Identity Federation for Kubernetes allows you to directly grant Kubernetes
ServiceAccounts access to Google resources, without the need to impersonate a Google
Service Account. This is done by granting the given IAM Roles directly to principals
of the form described above for impersonation. See [docs](https://cloud.google.com/iam/docs/workload-identity-federation-with-kubernetes#use-wlif).

Some specific GCP services do not support this method. See [docs](https://cloud.google.com/iam/docs/federated-identity-supported-services#list).

The `GET /computeMetadata/v1/instance/service-accounts/default/identity` API is not
supported by this method. If you plan to use this API, you must use the impersonation
method described above.

### Deploy `gke-metadata-server` in your cluster

#### Using the Helm Chart

A Helm Chart is available in the following [Helm OCI Repository](https://helm.sh/docs/topics/registries/):

`ghcr.io/matheuscscp/gke-metadata-server-helm:{version}` (GitHub Container Registry)

See the Helm Values API in the file [`./helm/gke-metadata-server/values.yaml`](./helm/gke-metadata-server/values.yaml).
Make sure to specify at least the full name of the Workload Identity Provider.

#### Using the Timoni Module

If you prefer something newer and strongly typed, a Timoni Module is available in the following
[Timoni OCI Repository](https://timoni.sh/concepts/#artifact):

`ghcr.io/matheuscscp/gke-metadata-server-timoni:{version}` (GitHub Container Registry)

See the Timoni Values API in the files [`./timoni/gke-metadata-server/templates/config.tpl.cue`](./timoni/gke-metadata-server/templates/config.tpl.cue)
and [`./timoni/gke-metadata-server/templates/settings.cue`](./timoni/gke-metadata-server/templates/settings.cue).
Make sure to specify at least the full name of the Workload Identity Provider.

#### Using only the container image (with your own Kubernetes manifests)

Alternatively, you can write your own Kubernetes manifests and consume only the container image:

`ghcr.io/matheuscscp/gke-metadata-server:{version}` (GitHub Container Registry)

### Verify the OCI artifacts

For verifying the OCI artifacts above, use the [`cosign`](https://github.com/sigstore/cosign) CLI tool.

#### Verify the container image

For verifying the container image of a given release, download the
digest file `container-digest.txt` attached to the Github Release
and use it with `cosign`:

```bash
VERSION="x.y.z" # replace with the release version
cosign verify ghcr.io/matheuscscp/gke-metadata-server@$(cat container-digest.txt) \
  --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
  --certificate-identity=https://github.com/matheuscscp/gke-metadata-server/.github/workflows/release.yml@refs/tags/v${VERSION}
```

##### Automatic verification

If you are using Sigstore's *Policy Controller* for enforcing policies you can automate the image
verification using [Keyless Authorities](https://docs.sigstore.dev/policy-controller/overview/#configuring-keyless-authorities).

If you are using *Kyverno* for enforcing policies you can automate the image verification using
[Keyless Verification](https://kyverno.io/docs/writing-policies/verify-images/sigstore/#keyless-signing-and-verification).

#### Verify the Helm Chart

For verifying the OCI Helm Chart of a given release, download the
digest file `helm-digest.txt` attached to the Github Release
and use it with `cosign`:

```bash
VERSION="x.y.z" # replace with the release version
cosign verify ghcr.io/matheuscscp/gke-metadata-server-helm@$(cat helm-digest.txt) \
  --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
  --certificate-identity=https://github.com/matheuscscp/gke-metadata-server/.github/workflows/release.yml@refs/tags/v${VERSION}
```

##### Automatic verification

If you are using a *Flux* `OCIRepository` for deploying Helm Charts you can automate the verification using
[Keyless Verification](https://fluxcd.io/flux/components/source/ocirepositories/#verification).

#### Verify the Timoni Module

For verifying the OCI Timoni Module of a given release, download the
digest file `timoni-digest.txt` attached to the Github Release
and use it with `cosign`:

```bash
VERSION="x.y.z" # replace with the release version
cosign verify ghcr.io/matheuscscp/gke-metadata-server-timoni@$(cat timoni-digest.txt) \
  --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
  --certificate-identity=https://github.com/matheuscscp/gke-metadata-server/.github/workflows/release.yml@refs/tags/v${VERSION}
```

##### Automatic verification

If you are using the Timoni CLI to deploy Modules/Bundles you can automate the verification using
[Keyless Verification](https://timoni.sh/cue/module/signing/#sign-with-cosign-keyless).

## Further documentation

### Routing Modes

The Google libraries attempt to retrieve tokens from the hard-coded endpoint
`169.254.169.254:80`. In order to make this work, the emulator must intercept
the traffic from the client Pods to this endpoint. To solve this problem, the
emulator supports three routing modes: `eBPF`, `Loopback`, and `None`. See details of
each mode below. In all modes the *unencrypted* communication between the
emulator Pod and a client Pod never leaves the Node where they are both
running on. This is exactly how the native GKE implementation works.

The annotation or label `node.gke-metadata-server.matheuscscp.io/routingMode`
can be added to a Node in order to configure the routing mode of the emulator
in that particular Node. The default is `eBPF`.

If you are using similar tools or workload identity features of managed Kubernetes
services from other clouds, *this configuration may have a direct conflict with other
such tools or features.* It's a recurrent practice among cloud providers using this
hard-coded IP address and port to implement workload identity. If your use case requires,
the recommended solution is to use Node pooling/grouping features from your Kubernetes
provider to isolate the workloads that need to use the emulator as mentioned in the
[Usage](#usage) section. Remember to add the label
`iam.gke.io/gke-metadata-server-enabled: "true"` to such Nodes (or set `requireNodeLabel: false`)
and appropriate Node selectors/tolerations to your Pods.

#### `eBPF`

In this routing mode the emulator hooks an eBPF program to the `connect()` syscall.
The eBPF program looks at the destination address and, if it matches the hard-coded
address mentioned above, modifies the address to the actual address of the emulator.

This mode also attaches a `cgroup/sock_ops` program that records the cgroup ID
of every TCP-connecting task into a 4-tuple-keyed map at `TCP_CONNECT_CB`. The
emulator consults that map for hostNetwork pod requests and resolves the cgroup
ID to a pod UID by walking `/sys/fs/cgroup`, giving hostNetwork pods per-pod
identity (see [Pods running on the host network](#pods-running-on-the-host-network)).

This mode coexists with other eBPF tools that attach to `cgroup/connect4` —
notably Cilium, including with [`kubeProxyReplacement`](https://docs.cilium.io/en/stable/network/kubernetes/kubeproxy-free/)
enabled (which makes Cilium attach its own `cgroup/connect4` for service-VIP
redirection). Both programs are loaded under the same hook via the kernel's
multi-attach API (`BPF_LINK_CREATE`), and act on disjoint destinations:
Cilium rewrites ClusterIP-range destinations, this emulator rewrites
`169.254.169.254:80` only. Either ordering between the two programs
produces the same result. CI validates this coexistence on every PR
(`e2e (cilium-kpr)` matrix entry).

The advantage of this mode is that the emulator can bind to any port, not just port 80.
This is because we modify the destination address of connections targeting our endpoint
of interest, so we have the freedom to choose any port.

Note that in this mode the emulator will proxy connections that are not targeting
Google metadata to the `169.254.169.254:80` endpoint, i.e. any connections whose
first few bytes do not match `GET /computeMetadata/v1`. This is to allow other
environments that expose services on this endpoint to continue working properly.
For example, AWS EKS clusters expose the AWS Instance Metadata Service on this
endpoint. The emulator works fine in such environments without disrupting the
native metadata service due to this proxying feature. The emulator will not work
with environments exposing a service in this endpoint that *also identifies Pods
by their IP addresses*. This is because the source IP address the service will see
will be the Node IP address, and not the IP address of the client Pod. This is
not the case for AWS EKS, since the AWS Instance Metadata Service does not identify
Pods by their IP addresses, so the emulator works fine in AWS EKS clusters.

#### `Loopback`

In this routing mode the emulator adds the hard-coded address mentioned above to the
`lo` interface of the Node with the label `lo:gke-md-sv` and the *host* scope. This
way the emulator can bind to the hard-coded address and port 80, and the traffic from
the client Pods to this address will be routed to the emulator.

This mode does not clash with other eBPF-based tools running in the cluster, but it
has the disadvantage of not being able to bind to any port, only port 80. And for
this to work properly the emulator Pods need to run on the host network, so they
occupy port 80 in the network namespace of the Node.

#### `None`

In this routing mode the emulator does not perform any network routing or traffic
interception. Instead, it relies on the environment variables defined inside the
Google libraries to redirect metadata requests to the emulator. Note that support
for the environment variables below may vary between different Google libraries.
The configuration of your Pods should include:

```yaml
spec:
  containers:
  - name: your-app
    env:
    - name: HOST_IP
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP
    - name: GKE_METADATA_SERVER_PORT
      value: "16321"
    - name: GCE_METADATA_HOST
      value: "$(HOST_IP):$(GKE_METADATA_SERVER_PORT)"
    - name: GCE_METADATA_ROOT
      value: "$(HOST_IP):$(GKE_METADATA_SERVER_PORT)"
    - name: GCE_METADATA_IP
      value: "$(HOST_IP):$(GKE_METADATA_SERVER_PORT)"
```

The port value must match the port configured on the emulator DaemonSet (default: 16321).
This approach avoids conflicts with other tools that could be attaching eBPF programs
to the `connect()` syscall or using the well-known IP address and port hard-coded in
the Google libraries, but requires a bit of per-Pod configuration.

### The `metadata.google.internal` DNS record

If you observe DNS lookup errors for `metadata.google.internal` in your Pods,
you can add to them the following DNS configuration:

```yaml
spec:
  hostAliases:
  - hostnames: [metadata.google.internal]
    ip: 169.254.169.254
```

Not all Google libraries hit `metadata.google.internal`, some just hit
`169.254.169.254` directly. The `gcloud` CLI, for example, needs this
DNS configuration to work properly. See this [example](./testdata/pod-gcloud.yaml).

If your cluster supports CoreDNS's `coredns-custom` ConfigMap, like k3s or AKS,
you can alternatively set `dns.provider` to `CoreDNSCustom` in the Helm Chart
or Timoni Module. When set, this option applies a ConfigMap called
`coredns-custom` in the `kube-system` namespace that instructs CoreDNS to
resolve `metadata.google.internal` to `169.254.169.254` cluster-wide,
eliminating the need for per-Pod `hostAliases` configuration.

> **Warning:** Enabling `CoreDNSCustom` will **overwrite** any existing
> `coredns-custom` ConfigMap in the `kube-system` namespace. If you already
> have custom CoreDNS entries in that ConfigMap, they will be lost. In that
> case, manually add the `metadata.override` entry to your existing ConfigMap
> instead of using this option.

### Node initialization

The emulator Pods have toleration for any taints, so they will get scheduled earlier
on the Nodes. This may be enough for ensuring the emulator is running before client
Pods attempt to get credentials from them. But if you observe issues on your Pods
due to this race condition, you can configure your Nodes to start with a taint that
will prevent your Pods from being scheduled on them until the emulator has fully
initialized. You can choose any key you want for the taint, as long as it has the
prefix `node.gke-metadata-server.matheuscscp.io`. The emulator will remove any such
taints from the Node after successfully initializing. See the taints and tolerations
docs [here](https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/).

### Limitations and Security Risks

#### Pod identification

The strategy is picked per request by `(routing mode, pod kind)`. There is
no fallback between strategies — each combination has exactly one path.

| routing mode | non-hostNetwork pod | hostNetwork pod |
|---|---|---|
| `eBPF` | kernel attestation (sockops 4-tuple → cgroup ID → pod UID) | same |
| `Loopback` | source IP → `GetByIP` (node-scoped) | netlink SOCK_DIAG → `/proc/<pid>/cgroup` → pod UID |
| `None` | source IP → `GetByIP` (node-scoped) | netlink SOCK_DIAG → `/proc/<pid>/cgroup` → pod UID |

In `eBPF` mode every pod is identified by **kernel attestation** — the
source IP is not consulted at all. In `Loopback` and `None` modes, regular
pods are identified by their source IP (pod IPs are unique within a Node
and `GetByIP` is node-scoped, so this is sound), while hostNetwork pods —
which share the Node's IP — fall back to the netlink-based attestation
chain. See the [host-network section](#pods-running-on-the-host-network)
below for the kernel-attestation details.

The trust root for source IP is the kernel's report on the TCP connection
— never any HTTP-level header (e.g. `X-Forwarded-For`).

If an attacker can perform source-IP impersonation in your cluster (e.g.
[ARP spoofing](https://cloud.hacktricks.xyz/pentesting-cloud/kubernetes-security/kubernetes-network-attacks)),
the source-IP strategy used by `Loopback`/`None` for non-hostNetwork pods
can be exploited to steal credentials. **Please evaluate the risk of such
attacks in your cluster before choosing this tool.** The attack referenced
typically requires Pods running with very high privileges, which should
normally be blocked in production clusters via PSA, [Gatekeeper](https://github.com/open-policy-agent/gatekeeper)
or equivalent policy. The kernel-attestation paths (`eBPF` for any pod,
netlink for hostNetwork pods) do not depend on source IP and are not
affected by ARP spoofing.

#### Pods running on the host network

Pods with `spec.hostNetwork: true` share the Node's IP address, so the
source IP the emulator sees on the connection is not unique to the pod.
Both routing-mode-specific kernel-attestation paths are described below;
the `eBPF` chain is the same path used for *every* pod in `eBPF` mode
(non-hostNetwork pods are also resolved through it), and the netlink
chain is only used for hostNetwork pods in `Loopback` and `None` modes.

- In `eBPF` routing mode, a `cgroup/sock_ops` BPF program records the
  connecting task's cgroup ID into a 4-tuple-keyed map at TCP connect time.
  The emulator looks up the 4-tuple from the accepted connection, walks
  `/sys/fs/cgroup` to find the matching directory, and extracts the pod UID
  from the kubelet's `pod<UID>` cgroup naming convention.
- In `Loopback` and `None` routing modes, the emulator queries the host's
  socket table via `NETLINK_SOCK_DIAG` to resolve the 4-tuple to its socket
  inode, walks `/proc/<pid>/fd` to find the owning PID, then reads
  `/proc/<pid>/cgroup` for the same `pod<UID>` extraction.

Both paths produce a kubernetes pod UID without trusting source IP or any
HTTP-level information. The pod's regular ServiceAccount (configured via
`spec.serviceAccountName`) is then used; there is **no** node-level shared
ServiceAccount.

Hard requirements: a Linux kernel with cgroup v2 unified hierarchy and BPF
cgroup-hook support (in practice any kernel from the last few years —
`bpf_get_current_cgroup_id` has been available since 4.18). No specific
CNI is required. CI exercises three configurations on every PR:
`cilium-default` (Cilium 1.19.x with default values), `cilium-kpr` (Cilium
with `kubeProxyReplacement=true`, which makes Cilium attach its own
`cgroup/connect4` program alongside ours), and `kindnet` (kind's default
CNI, no Cilium).

Sandboxed container runtimes (gVisor, Kata) are unsupported because their
userspace kernels are not visible to host BPF programs and not represented
in the host's `/proc`.

#### Token Cache

When the emulator is configured to cache tokens, the issued Google OAuth 2.0 Access and
OpenID Connect Identity Tokens are cached and returned to client Pods on every request
until their expiration.

This means that even if you revoke the required permissions for the Kubernetes ServiceAccount
to issue those tokens, the client Pods will still get those tokens from the emulator until
they expire. This is a limitation of how the permissions are evaluated: they are evaluated
only when the tokens are issued, which is what caching tries to avoid. If your use case
requires immediate revocation of permissions, one alternative is to disable token caching.
The Helm Chart and Timoni Module distributed here have options to disable token caching, look
at their respective values APIs. Another alternative is manually deleting all the Pods of
the emulator DaemonSet:

```bash
kubectl delete pods -l app=gke-metadata-server -n kube-system
```

New Pods will be created by the DaemonSet and they will not have the cached tokens.

The emulator implements an 80% refresh rule (similar to kubelet for ServiceAccount token
rotation) and supports configurable maximum token duration limits. The 80% rule means
tokens are refreshed when they have consumed 80% of their original lifetime. The maximum
token duration setting allows capping the effective duration of cached tokens (default: 1 hour).
This can be configured through the Helm Chart and Timoni Module values APIs.

With the default configuration, tokens expire in at most one hour.

## Disclaimer

This project was not created by Google. Enterprise support from Google is
not available. **Use this tool at your own risk.**
*(But please do feel free to report bugs and CVEs, request help, new features and
[contribute](https://github.com/matheuscscp/gke-metadata-server/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22).)*

Furthermore, this tool is *not necessary* for using GCP Workload Identity Federation
inside non-GKE Kubernetes clusters. This is just a facilitator. Kubernetes and GCP
Workload Identity Federation work together by themselves. This tool just makes your
Pods need much less configuration to use GCP Workload Identity Federation for Kubernetes,
by making the configuration as close as possible to how Workload Identity Federation
for GKE is configured.

gke-metadata-server

Explore Similar Repositories

Was this content helpful?

Recent Feedback

Download README