If you have ever setup a Kubernetes cluster and had to add a Load Balancer in front of the API server or simply forgot to reserve a static IP, you would have have faced the challenge where the API server’s certificate does not have the new IP or the hostname. As the external IP address of the control plane has now changed, kubectl
will complain about the certificate not being authorized for the new hostname or IP address.
A quick solution to this is to use --insecure-skip-tls-verify
either as a command line argument to kubectl
or add the configuration to the kubeconfig permanently. This of course skips TLS verification and is considered insecure in the long run.
This post shows how you can add a new IP address or hostname to the API server certificate Subject Alternate Name (SAN) field to allow continued secure access.
Assuming you have updated the kubeconfig
to point the cluster server variable to the new IP or hostname, you can follow these steps to update the API server configuration.
Obtain the kubeadm
configuration file from the cluster and save it locally
kubectl -n kube-system get configmap kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' --insecure-skip-tls-verify > kubeadm.yaml
Open the file in a local text editor and and find the certSANs
section under apiServer
. If this is not there, then it must be created. Here’s an example of what it looks like.
apiServer:
certSANs:
- "10.10.10.100"
- "kubernetes.default"
extraArgs:
authorization-mode: Node,RBAC
timeoutForControlPlane: 4m0s
Add the new IP address or hostname to the certSANs
section
apiServer:
certSANs:
- "10.10.10.100"
- "kubernetes.default"
- "new-hostname-kubernetes"
- "X.X.X.X"
extraArgs:
authorization-mode: Node,RBAC
timeoutForControlPlane: 4m0s
SSH to the API server (control plane) and move the old certificates to another folder so that kubeadm
can recreate new ones:
mv /etc/kubernetes/pki/apiserver.{crt,key} ~
Use kubeadm
to generate new apiserver certificates:
kubeadm init phase certs apiserver --insecure-skip-tls-verify --config kubeadm.yaml
Now stop and restart the kubeapiserver container using docker or critools
Run docker ps | grep kube-apiserver | grep -v pause
to get the container ID for the container running the Kubernetes API server
Run docker kill <containerID>
to kill the container.
The kubelet
will automatically restart the container, which will pick up the new certificates.
If everything is working as expected, upload the kubeadm configuration for future upgrades
kubeadm init phase upload-config kubeadm --config kubeadm.yaml
kubectl get nodes