Let's Encrypt is the right answer for anything the internet talks to. It is the wrong answer for a database inside your cluster, a gRPC service that only other pods call, or a webhook. Those names aren't public, so no public CA will sign them, and you don't want internal hostnames in public Certificate Transparency logs anyway.
The usual fix is a private certificate authority. cert-manager can create one, sign certificates from it and renew them. The part people skip is trust: a certificate from a private CA is useless until every client has the root certificate. That is the job of trust-manager, a sibling project that copies a CA bundle into the namespaces that need it.
This guide builds the whole thing on a test cluster: a self-signed root, a CA ClusterIssuer, a leaf certificate for a service, and a trust bundle that pods can mount. It also covers the two places this setup tends to break later, which are root expiry and root rotation.
Before you start
You need a cluster, kubectl, helm and openssl on your machine. A local kind cluster is enough. The commands below install cert-manager v1.21.3 and trust-manager v0.26.0, both into the cert-manager namespace.
The cert-manager docs say plainly that SelfSigned and CA issuers are for people who are ready to run a PKI. That warning is fair. Nothing here is hard, but a private CA comes with chores: tracking expiry, planning rotation and deciding who can read the CA key. This guide points those out as it goes.
Step 1: install cert-manager and trust-manager
Install cert-manager first, with its CRDs, from the OCI chart. Then install trust-manager into the same namespace. trust-manager doesn't need cert-manager to run, but its Helm chart uses cert-manager to issue the TLS certificate for its own webhook, so the order matters here.
helm install cert-manager oci://quay.io/jetstack/charts/cert-manager \
--namespace cert-manager \
--create-namespace \
--version v1.21.3 \
--set crds.enabled=true
helm upgrade trust-manager oci://quay.io/jetstack/charts/trust-manager \
--install \
--namespace cert-manager \
--version v0.26.0 \
--wait
trust-manager has a setting called the trust namespace (Helm value app.trust.namespace). It is the only namespace where trust-manager reads source Secrets, and it defaults to cert-manager. This guide keeps the default. In production the trust-manager docs suggest a dedicated namespace, so fewer people and controllers can change what your cluster trusts.
Step 2: create the root CA
A CA issuer needs a CA certificate and its private key in a Secret. You could make one with openssl and upload it, but cert-manager can make it for you. A SelfSigned issuer signs a certificate with that certificate's own key, which is exactly what a root CA is.
The manifest below creates a SelfSigned ClusterIssuer, then a Certificate with isCA: true that uses it. The Certificate lives in cert-manager because a ClusterIssuer looks for its Secret in the cluster resource namespace, which is cert-manager by default.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: selfsigned-bootstrap
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: internal-root-ca
namespace: cert-manager
spec:
isCA: true
commonName: internal-root-ca
subject:
organizations:
- Example Internal
secretName: internal-root-ca
duration: 87600h
privateKey:
algorithm: ECDSA
size: 256
issuerRef:
name: selfsigned-bootstrap
kind: ClusterIssuer
group: cert-manager.io
Three fields deserve a comment. isCA: true sets the basic constraints extension and adds the cert sign usage on its own. The subject matters because a self-signed certificate's issuer DN is copied from its subject. Leave the subject empty and the issuer DN is empty too, which RFC 5280 forbids, and cert-manager warns about it with a BadConfig event.
duration is the one people forget. Without it, cert-manager uses its default of 90 days, which is fine for a leaf and far too short for a root. Here the root lasts ten years. Pick a number you can actually plan around.
Apply it and check that the Certificate is ready:
kubectl apply -f root-ca.yaml
kubectl get certificate -n cert-manager internal-root-ca
Step 3: create the CA ClusterIssuer
Now point a CA issuer at the Secret the Certificate produced. A ClusterIssuer lets any namespace request certificates from this CA. If only one team should use it, make it a namespaced Issuer in their namespace and create the root Certificate there instead.
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: internal-ca
spec:
ca:
secretName: internal-root-ca
kubectl apply -f internal-ca.yaml
kubectl get clusterissuer internal-ca
The issuer shows READY True once cert-manager has loaded the key pair. If it stays false, kubectl describe clusterissuer internal-ca usually says the Secret is missing, which almost always means it sits in a namespace other than the cluster resource namespace.
Keep in mind who can read that Secret. Anyone with get on Secrets in cert-manager can sign certificates your whole cluster will trust. Check your RBAC before you hand this CA to real workloads.
Step 4: issue a certificate for a service
With the issuer in place, a service owner writes a normal Certificate. Say you have a service called orders in the shop namespace. Its clients will connect to the in-cluster DNS name, so that name goes into dnsNames.
apiVersion: v1
kind: Namespace
metadata:
name: shop
labels:
trust.example.com/internal-ca: "enabled"
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: orders-tls
namespace: shop
spec:
secretName: orders-tls
dnsNames:
- orders.shop.svc
- orders.shop.svc.cluster.local
issuerRef:
name: internal-ca
kind: ClusterIssuer
group: cert-manager.io
The namespace label isn't used by cert-manager. trust-manager will use it in the next step to decide where the trust bundle goes.
After a few seconds the orders-tls Secret holds tls.crt, tls.key and ca.crt. This leaf has no duration, so it lasts 90 days and cert-manager renews it when a third of its lifetime is left. Your server has to pick up the new file, either by watching it or by restarting.
You can check the chain on your own machine before anything uses it:
kubectl get secret -n cert-manager internal-root-ca \
-o jsonpath='{.data.tls\.crt}' | base64 -d > root.pem
kubectl get secret -n shop orders-tls \
-o jsonpath='{.data.tls\.crt}' | base64 -d > orders.pem
openssl verify -CAfile root.pem orders.pem
openssl x509 -in orders.pem -noout -issuer -subject -ext subjectAltName
openssl verify should print orders.pem: OK. If it doesn't, stop here. The bundle in the next step won't fix a chain that doesn't verify.
Step 5: copy the root into its own ConfigMap
The quickest way to build a bundle is to point trust-manager straight at the internal-root-ca Secret. The trust-manager quick start does exactly that, with a warning that it isn't safe in production. The reason is rotation. When the root Secret changes, trust-manager updates every bundle at once, and the old root disappears from every client in the same moment. Any server still presenting a leaf signed by the old root stops being trusted.
So copy the root into a ConfigMap that you control. trust-manager reads it from there, and trust changes only when you edit it.
kubectl create configmap internal-root-ca-trust \
--namespace cert-manager \
--from-file=root.pem=root.pem
The ConfigMap has to live in the trust namespace, cert-manager in this setup. Use the root itself and nothing else. The trust-manager docs warn against putting intermediates in a trust store, because an intermediate that clients trust directly becomes a root you can no longer rotate quietly.
Step 6: create the Bundle
A Bundle is a cluster-scoped trust-manager resource. It lists sources, builds one PEM file from them, and writes it to a ConfigMap with the Bundle's name in each matching namespace.
apiVersion: trust.cert-manager.io/v1alpha1
kind: Bundle
metadata:
name: internal-trust
spec:
sources:
- configMap:
name: internal-root-ca-trust
key: root.pem
target:
configMap:
key: trust-bundle.pem
namespaceSelector:
matchLabels:
trust.example.com/internal-ca: "enabled"
This bundle contains only your private root. If the same clients also call public HTTPS endpoints, add - useDefaultCAs: true to the sources to include trust-manager's default public CA package. If you do, keep that package up to date, since it is how fixes such as the eMudhra distrust reach your pods.
The namespaceSelector keeps the bundle out of namespaces that don't need it. Without one, trust-manager writes it to every namespace. Apply the Bundle and check that it synced:
kubectl apply -f bundle.yaml
kubectl get bundle internal-trust
kubectl get configmap -n shop internal-trust \
-o jsonpath='{.data.trust-bundle\.pem}'
The SYNCED column should read True, and the ConfigMap in shop should contain your root.
Step 7: mount the bundle in clients
A client pod mounts the ConfigMap like any other. Go's standard library and the curl command line tool both read the SSL_CERT_FILE environment variable, as do many other OpenSSL-based clients, so you can often point a client at the bundle without changing its code.
apiVersion: v1
kind: Pod
metadata:
name: trust-test
namespace: shop
spec:
containers:
- name: curl
image: curlimages/curl:8.11.1
command: ["sleep", "3600"]
env:
- name: SSL_CERT_FILE
value: /etc/internal-trust/trust-bundle.pem
volumeMounts:
- name: internal-trust
mountPath: /etc/internal-trust
readOnly: true
volumes:
- name: internal-trust
configMap:
name: internal-trust
Once the orders server serves the orders-tls certificate on port 443, kubectl exec -n shop trust-test -- curl https://orders.shop.svc will succeed without --insecure. In Go, SSL_CERT_FILE overrides the system's default certificate file, so a bundle that holds only your root can stop that client from trusting public sites. That is one more reason to add useDefaultCAs when a client also talks to the outside world.
Plan for root expiry and rotation
The root Certificate is still a cert-manager Certificate, so cert-manager renews it. With a ten-year duration and no renewBefore, renewal happens when a third of the lifetime is left, a bit over three years before expiry. Since v1.18 the default privateKey.rotationPolicy is Always, so that renewal makes a new key, which means a new root.
Two things then happen that catch people out. The CA issuer starts signing with the new root, but existing leaves aren't reissued, because changing the CA Secret doesn't trigger reissuance. And if your Bundle pointed straight at the Secret, every client would drop the old root at once. The copied ConfigMap from step 5 is what saves you here.
The rollover looks like this. Add the new root to internal-root-ca-trust next to the old one, so clients trust both. Then reissue every leaf so it's signed by the new root. cmctl renew does this on demand, one Certificate at a time or across a namespace. When nothing presents a leaf from the old root, remove it from the ConfigMap.
kubectl get secret -n cert-manager internal-root-ca \
-o jsonpath='{.data.tls\.crt}' | base64 -d > root-new.pem
cat root.pem root-new.pem > both.pem
kubectl create configmap internal-root-ca-trust \
--namespace cert-manager \
--from-file=root.pem=both.pem \
--dry-run=client -o yaml | kubectl apply -f -
cmctl renew --namespace shop --all
The CA issuer also doesn't check whether a leaf outlives the CA that signed it. Watch the expiry of every certificate in the chain, the root included, with whatever alerting you already use for leaf certificates. For the root, also watch status.renewalTime on its Certificate, because that date, not the expiry, is when a new root appears and the rollover above has to start.