A kubeadm cluster runs fine for a year, and then one morning kubectl answers with x509: certificate has expired or is not yet valid. Nothing changed in the cluster. The control plane client certificates that kubeadm generated at install time simply reached their end date.
This guide shows how to see that date coming, renew the certificates by hand, make the control plane pick them up, and check that the new ones are really in use. It also covers the kubelet certificates, which follow different rules, and how to stop this from becoming a yearly fire drill.
Why kubeadm certificates expire
kubeadm builds a full PKI under /etc/kubernetes/pki when you run kubeadm init. It creates three certificate authorities (ca, etcd-ca and front-proxy-ca) and a set of leaf certificates signed by them: the API server serving certificate, the client certificates the API server uses to reach etcd and the kubelets, the etcd server and peer certificates, and the client certificates embedded in admin.conf, super-admin.conf, controller-manager.conf and scheduler.conf.
The upstream defaults are one year for leaf certificates and ten years for the CAs. You can change them with the certificateValidityPeriod and caCertificateValidityPeriod fields of the kubeadm ClusterConfiguration (default 8760h and 87600h). Most clusters keep the defaults, so the leaf certificates expire about a year after install or after the last renewal.
A short lifetime is a reasonable default. A stolen client certificate stays useful only until it expires, and Kubernetes has no certificate revocation. The catch is that someone has to renew them, and kubeadm only does it for you in one situation, which we will get to.
Step 1: check when the certificates expire
Run this on a control plane node. It reads the files on disk and does not need a working API server.
sudo kubeadm certs check-expiration
The output has two tables. The first lists the leaf certificates and the kubeconfig files, with the expiry date, the time left and the CA that signed each one. The second lists the CAs. The upstream docs show output like this for a fresh cluster:
CERTIFICATE EXPIRES RESIDUAL TIME CERTIFICATE AUTHORITY EXTERNALLY MANAGED
admin.conf Dec 30, 2020 23:36 UTC 364d no
apiserver Dec 30, 2020 23:36 UTC 364d ca no
apiserver-etcd-client Dec 30, 2020 23:36 UTC 364d etcd-ca no
apiserver-kubelet-client Dec 30, 2020 23:36 UTC 364d ca no
controller-manager.conf Dec 30, 2020 23:36 UTC 364d no
etcd-healthcheck-client Dec 30, 2020 23:36 UTC 364d etcd-ca no
etcd-peer Dec 30, 2020 23:36 UTC 364d etcd-ca no
etcd-server Dec 30, 2020 23:36 UTC 364d etcd-ca no
front-proxy-client Dec 30, 2020 23:36 UTC 364d front-proxy-ca no
scheduler.conf Dec 30, 2020 23:36 UTC 364d no
CERTIFICATE AUTHORITY EXPIRES RESIDUAL TIME EXTERNALLY MANAGED
ca Dec 28, 2029 23:36 UTC 9y no
etcd-ca Dec 28, 2029 23:36 UTC 9y no
front-proxy-ca Dec 28, 2029 23:36 UTC 9y no
Look at the EXTERNALLY MANAGED column. A yes means kubeadm found a CA certificate without its private key, so it cannot sign anything with that CA. If you run in that external CA mode, kubeadm cannot renew those certificates and you have to sign new ones with your own CA. Everything below assumes the CA keys are on the node, which is the default.
You will also notice that kubelet.conf is missing from the list. That is on purpose, and we come back to it later.
Step 2: back up the current PKI
Renewal overwrites the certificate and key files in place. If something goes wrong halfway, you want the old files to go back to. A plain copy of /etc/kubernetes is enough.
sudo cp -a /etc/kubernetes "/root/kubernetes-backup-$(date +%F)"
This copy holds every CA private key in the cluster. Keep it on the node or in a place with the same protection, and delete it once the renewal is done and verified.
Step 3: renew the certificates
kubeadm certs renew can renew one certificate by name, or all of them with the all subcommand. Renewing everything at once keeps the expiry dates aligned, which makes the next renewal easier to plan.
sudo kubeadm certs renew all
kubeadm prints one line per certificate as it renews it, and a MISSING! line for any file that is not present on this node, which it then skips. It finishes with a reminder:
Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.
Two details from the kubeadm source are worth knowing. First, renewal is unconditional. kubeadm issues a new certificate even if the old one has 300 days left, so running the command early is safe and resets the clock. Second, kubeadm takes the subject, organization and SANs from the existing certificate, not from the kubeadm-config ConfigMap. Renewal will not add a new SAN for you. If you need one, that is a separate task where you regenerate the API server certificate.
The new certificates get the validity set in certificateValidityPeriod, or one year if you never set it.
If your cluster has several control plane nodes, every node has its own copy of these files. You have to run the renewal on each control plane node. Do them one at a time, finishing the next two steps on a node before you move to the next one, so the cluster always has a healthy API server.
Step 4: restart the control plane static pods
The new files are on disk, but the running processes still hold the old certificates in memory. Not every component reloads certificates on its own, so you need to restart the API server, the controller manager, the scheduler and etcd.
These run as static pods. The kubelet manages them directly from the manifests in /etc/kubernetes/manifests, so kubectl delete pod does not restart them for real. The upstream method is to move a manifest out of that directory, let the kubelet stop the pod, and move it back. The kubelet checks the directory every 20 seconds by default (the fileCheckFrequency field of its configuration), so each move needs a wait of at least that long.
The loop below does this one component at a time. It moves the manifests to a directory outside the static pod path, so the kubelet ignores them while they are away.
sudo mkdir -p /etc/kubernetes/manifests-paused
for c in kube-apiserver kube-controller-manager kube-scheduler etcd; do
echo "Restarting ${c}"
sudo mv "/etc/kubernetes/manifests/${c}.yaml" /etc/kubernetes/manifests-paused/
sleep 30
sudo mv "/etc/kubernetes/manifests-paused/${c}.yaml" /etc/kubernetes/manifests/
sleep 30
done
sudo rmdir /etc/kubernetes/manifests-paused
On a single control plane node the API server is down for about a minute while its manifest is away. Running workloads keep running, since the kubelets and containers on the worker nodes do not need the API server to keep existing pods alive. You just cannot schedule or change anything during that window, so pick a quiet time.
Check that each container came back with a fresh start time:
sudo crictl ps --name 'kube-apiserver|kube-controller-manager|kube-scheduler|etcd'
The CREATED column should show a few minutes for all four.
Step 5: refresh your admin kubeconfig
The renewal rewrote /etc/kubernetes/admin.conf with a new client certificate. Many people copied that file to ~/.kube/config when they built the cluster, and the copy still holds the old certificate. Copy it again:
sudo cp -i /etc/kubernetes/admin.conf "$HOME/.kube/config"
sudo chown "$(id -u):$(id -g)" "$HOME/.kube/config"
If you handed out copies of admin.conf to other people or CI jobs, they all break when the old certificate expires. This is a good moment to replace them with per-user credentials. kubeadm kubeconfig user generates a kubeconfig with its own client certificate and lifetime, and RBAC can then limit what it does. admin.conf and super-admin.conf are meant for break-glass use, not daily work.
Step 6: confirm the new certificates are in use
Files on disk can lie. What matters is the certificate the API server presents. Ask it directly with openssl and compare the end date with the one in the file:
echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -enddate
sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -enddate
Both lines should show the same notAfter date, about a year from today. If the first one still shows the old date, the API server did not restart. Go back to step 4.
Then run kubeadm certs check-expiration once more, and a quick kubectl get nodes to make sure your refreshed kubeconfig works.
What about the kubelet certificates
kubelet.conf is not in the renewal list because kubeadm does not manage it. kubeadm sets up the kubelet with automatic client certificate rotation. The kubelet requests a new client certificate from the API server as the old one nears expiry, stores it under /var/lib/kubelet/pki, and kubelet.conf points at the kubelet-client-current.pem symlink. As long as the controller manager can sign certificates, you have nothing to do.
You can check the current kubelet client certificate on any node:
sudo openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -noout -enddate
If that rotation ever fails, for example because a node stayed powered off past the expiry date, the API server logs x509: certificate has expired or is not yet valid errors for that node. The kubeadm troubleshooting docs describe the fix: generate a new kubelet.conf with kubeadm kubeconfig user --org system:nodes --client-name system:node:<node-name> on a control plane node, copy it to the broken node and restart the kubelet. Once the kubelet has written a fresh kubelet-client-current.pem, edit kubelet.conf to point at that file instead of the embedded certificate, and restart the kubelet again so rotation works from then on.
The kubelet serving certificate (the one metrics-server sees on port 10250) is different again. By default kubeadm leaves it self-signed. If you enable serverTLSBootstrap: true in the kubelet configuration, the kubelet requests a signed one instead, but the controller manager does not approve those requests automatically. You have to approve them with kubectl certificate approve, or run a controller that checks and approves them, and the same approval is needed every time they rotate.
Stop doing this by hand
kubeadm renews all the certificates it manages during a control plane upgrade. Both kubeadm upgrade apply and kubeadm upgrade node do it by default, and only skip it if you pass --certificate-renewal=false. Kubernetes ships a minor release roughly every four months, so a team that upgrades at least once a year never hits the expiry at all. In our experience that is the best fix. Clusters that hit expired certificates are almost always clusters that also stopped getting upgrades.
If you cannot upgrade that often, put the check on a schedule. A daily job that runs kubeadm certs check-expiration on each control plane node and alerts when the residual time drops below 30 days catches the problem with weeks to spare. And write the restart step from this guide into your runbook. Renewing the files is the easy part. Forgetting to restart the static pods is how a renewal that looked finished still ends in an outage a few weeks later.