# Cosign with GitLab CI, Cloud KMS and Flux

> A GitLab pipeline signs an image and a Flux OCI artifact of Kustomize manifests with a Cloud KMS key it reaches through Workload Identity Federation. Flux verifies the artifact with the public key before it applies anything.

Interactive graph: https://k8wiz.com/graphs/cosign-gitlab-ci-cloud-kms-flux-oci-verification

Two things get signed here: the container image, and the Kustomize manifests that deploy it, pushed to Artifact Registry as a Flux OCI artifact. Both are OCI digests, so cosign signs them the same way, with one asymmetric key in Cloud KMS.

The private key never leaves KMS. GitLab doesn't hold a secret either: the job trades its own OIDC token for short-lived Google credentials through Workload Identity Federation, and only pipelines on protected refs of one project may use the key. The cluster gets the public key only. Flux checks the manifests with it, and an admission controller checks the images.

## Resources

### CI job release (.gitlab-ci.yml)

outside the cluster

Runs on protected refs only. It exchanges its GitLab ID token for Google credentials, builds and signs the image, pins the image digest in the Kustomize overlay, pushes the manifests with `flux push artifact` and signs that digest too. It holds no key and no service account JSON.

### OIDC provider gitlab.com

outside the cluster

Issues the job an ID token for the audience named in `id_tokens`. Its claims say which project, ref and pipeline the job belongs to, for example `project_path: acme/shop`, `ref: main`, `ref_protected: "true"`, and `sub: project_path:acme/shop:ref_type:branch:ref:main`.

### Workload Identity pool gitlab provider

outside the cluster

Trusts tokens from `https://gitlab.com`, maps claims to attributes and rejects tokens that fail the attribute condition, such as `assertion.project_path == "acme/shop" && assertion.ref_protected == "true"`. This condition decides who can sign. Google's STS then trades the token for a short-lived federated token.

### Service account cosign-signer@acme.iam

outside the cluster

The identity the job impersonates. It has `roles/cloudkms.signerVerifier` on the one signing key and write access to the Artifact Registry repository, nothing else. The federated principal for `acme/shop` has `roles/iam.workloadIdentityUser` on it.

### KMS key release/cosign (EC P-256)

outside the cluster

An asymmetric signing key, `ec-sign-p256-sha256`. cosign sends it a digest to sign and gets a signature back; the private key can't be exported. Audit logs record every signing call. Rotation adds a key version; old versions can be disabled to stop new signatures.

### Image shop@sha256:aaa…

made by a build or publishing tool

The container image, built and pushed by the job. Its digest is what gets signed and what the manifests pin.

### Signature image signature

made by a build or publishing tool

Pushed by `cosign sign --key gcpkms://…`. With cosign 3 it's a Sigstore bundle attached to the image through the referrers API. It holds a signature that only the KMS key's public half verifies.

### Flux artifact shop-manifests@sha256:bbb…

made by a build or publishing tool

The built `deploy/production` directory, packed by `flux push artifact` with Flux's media types and the Git source and revision as annotations. The tag `production` points at it. Because CI ran `kustomize edit set image` first, the Deployment inside names the image by digest.

### Signature manifests signature

made by a build or publishing tool

The same kind of signature, over the manifests artifact's digest. It's what Flux checks before any YAML reaches the cluster.

### Secret cosign-pub

you write it, v1, namespace flux-system

The public half of the KMS key, exported with `cosign public-key --key gcpkms://…`. Flux accepts a signature from any key in the Secret whose name ends in `.pub`, so during a rotation you list both versions.

### OCIRepository shop-manifests

you write it, source.toolkit.fluxcd.io/v1, namespace flux-system

Follows the `production` tag. `provider: gcp` makes source-controller pull with its GKE Workload Identity instead of a registry secret. `verify` makes it check the cosign signature with the public key before it produces an artifact.

### Flux controller source-controller

comes with an add-on, namespace flux-system

Every interval, resolves `production` to a digest. When it changes, it verifies the signature for that digest, then downloads and unpacks the layer. A failed check leaves the OCIRepository not ready with the verification error, and the last good artifact stays in use.

### Kustomization shop

you write it, kustomize.toolkit.fluxcd.io/v1, namespace flux-system

Applies the verified artifact. It only ever sees manifests that passed the signature check.

### Deployment shop

created by a controller, apps/v1, namespace shop

Applied by the Kustomization from the artifact. Its image is the digest CI signed, so a tag moved in the registry can't change what runs.

### Admission check Kyverno or Binary Authorization

comes with an add-on

Flux verified the YAML, not the images it names. An admission controller checks the image signatures when Pods are created: a Kyverno `ImageValidatingPolicy` with a cosign key attestor holding the same public key, or GKE Binary Authorization with an attestor backed by the same Cloud KMS key.

## How the resources connect

- CI job release (.gitlab-ci.yml) → OIDC provider gitlab.com: id_tokens (traffic)
- CI job release (.gitlab-ci.yml) → Workload Identity pool gitlab provider: exchange ID token (traffic)
- Workload Identity pool gitlab provider → OIDC provider gitlab.com: trusts issuer, checks claims (references)
- Workload Identity pool gitlab provider → Service account cosign-signer@acme.iam: impersonate (references)
- Service account cosign-signer@acme.iam → KMS key release/cosign (EC P-256): signerVerifier (references)
- CI job release (.gitlab-ci.yml) → KMS key release/cosign (EC P-256): asymmetricSign (traffic)
- CI job release (.gitlab-ci.yml) → Image shop@sha256:aaa…: build, push (creates)
- CI job release (.gitlab-ci.yml) → Signature image signature: cosign sign --key (creates)
- Signature image signature → Image shop@sha256:aaa…: subject (references)
- CI job release (.gitlab-ci.yml) → Flux artifact shop-manifests@sha256:bbb…: flux push artifact (creates)
- Flux artifact shop-manifests@sha256:bbb… → Image shop@sha256:aaa…: pins digest (references)
- CI job release (.gitlab-ci.yml) → Signature manifests signature: cosign sign --key (creates)
- Signature manifests signature → Flux artifact shop-manifests@sha256:bbb…: subject (references)
- Secret cosign-pub → KMS key release/cosign (EC P-256): public key of (references)
- OCIRepository shop-manifests → Secret cosign-pub: verify.secretRef (references)
- Flux controller source-controller → OCIRepository shop-manifests: reconciles (references)
- Flux controller source-controller → Signature manifests signature: verifies (traffic)
- Flux controller source-controller → Flux artifact shop-manifests@sha256:bbb…: pulls (provider: gcp) (traffic)
- Kustomization shop → OCIRepository shop-manifests: sourceRef (references)
- Kustomization shop → Deployment shop: applies (creates)
- Deployment shop → Image shop@sha256:aaa…: image@digest (references)
- Admission check Kyverno or Binary Authorization → Signature image signature: verifies (traffic)
- Admission check Kyverno or Binary Authorization → Deployment shop: admits its Pods (traffic)

## How it works

1. **Create the key and decide who may use it.** The signing key is an asymmetric `ec-sign-p256-sha256` key in Cloud KMS. The service account `cosign-signer` may sign with it, and only federated identities from `acme/shop` on protected refs may impersonate that service account. Those IAM bindings are the trust decision: anyone who can use the key can sign.
2. **The job proves which pipeline it is.** GitLab gives the job an ID token for the Workload Identity provider's audience. `gcloud iam workload-identity-pools create-cred-config` writes a credentials file that points at it, and Google's STS checks the issuer and the attribute condition before it hands out short-lived credentials for `cosign-signer`. A pipeline on an unprotected branch or another project gets nothing.
3. **Build the image and sign its digest.** The job pushes the image and reads its digest from the buildx metadata. `cosign sign --yes --key gcpkms://…` hashes what it signs locally, asks KMS to sign the hash, and pushes the signature next to the image. The private key stays in KMS.
4. **Pin the image in the manifests.** `kustomize edit set image shop=…@sha256:aaa…` writes the signed digest into the overlay. From now on, the manifests and the image are tied together: signing the manifests also fixes which image they deploy.
5. **Push the manifests as a Flux artifact and sign them.** `flux push artifact --output json` uploads the overlay and prints the repository and digest, which go straight to `cosign sign`. The job tags the artifact `production` last, after it's signed, so the tag never points at unsigned content.
6. **Give the cluster the public key.** `cosign public-key --key gcpkms://…` prints the public key. Store it as `cosign.pub` in a Secret in `flux-system`. The cluster never needs KMS access, so a compromised cluster can't sign anything.
7. **Flux verifies before it applies.** source-controller sees `production` move to a new digest, checks the signature against the keys in `cosign-pub`, and only then unpacks the artifact. If the check fails, the OCIRepository reports the error and the Kustomization keeps the last verified revision.
8. **The Kustomization applies the pinned Deployment.** The Kustomization applies what the verified artifact contains, including the Deployment with the image digest. `flux get sources oci` shows the revision and digest in use.
9. **Admission checks the image itself.** Flux's check covers the YAML. Kyverno or Binary Authorization checks the image signature with the same public key when the Pods are created, which also catches images deployed outside Flux.

## Common mistakes

- A Workload Identity attribute condition that only checks the GitLab group, or no condition at all, lets any project in it, or any GitLab user, sign with your key. Check `project_path` and `ref_protected`, and protect the branches and tags that release.
- Grant `roles/cloudkms.signerVerifier` on the key, not the project. A project-level grant lets the signer use every key in the project.
- cosign records key-based signatures in Sigstore's public Rekor log by default, which exposes registry paths and digests. Decide whether that's acceptable, and check your cosign version's flags if it isn't.
- Cosign 3 attaches signatures as Sigstore bundles through referrers. Sign one test artifact and confirm the OCIRepository verifies it with your Flux version before you switch production.
- Flux verifies the manifests artifact, not the images it references. Without an admission check, an unsigned image deployed with `kubectl` or another tool still runs.
- Removing the old public key from `cosign-pub` right after a rotation breaks the next reconcile of every artifact still signed with it. Keep both keys until all current artifacts are re-signed or replaced.

## The manifests you write

```yaml
apiVersion: v1
kind: Secret
metadata:
  name: cosign-pub
  namespace: flux-system
stringData:
  cosign.pub: |
    -----BEGIN PUBLIC KEY-----
    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE…
    -----END PUBLIC KEY-----
---
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: shop-manifests
  namespace: flux-system
spec:
  interval: 5m
  url: oci://europe-west1-docker.pkg.dev/acme/apps/shop-manifests
  ref:
    tag: production
  provider: gcp
  verify:
    provider: cosign
    secretRef:
      name: cosign-pub
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: shop
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: OCIRepository
    name: shop-manifests
  path: ./
  prune: true
```
