How to add a new hostname or IP address to a Standalone Kubernetes API server

The Problem

If you have ever setup a Kubernetes cluster and had to add a Load Balancer in front of the API server or simply forgot to reserve a static IP, you would have have faced the challenge where the API server’s certificate does not have the new IP or the hostname. As the external IP address of the control plane has now changed, kubectl will complain about the certificate not being authorized for the new hostname or IP address.

A quick solution to this is to use --insecure-skip-tls-verify either as a command line argument to kubectl or add the configuration to the kubeconfig permanently. This of course skips TLS verification and is considered insecure in the long run.

This post shows how you can add a new IP address or hostname to the API server certificate Subject Alternate Name (SAN) field to allow continued secure access.

Adding a new hostname or IP

Assuming you have updated the kubeconfig to point the cluster server variable to the new IP or hostname, you can follow these steps to update the API server configuration.

1. Obtain the kubeadm configuration file from the cluster and save it locally

   kubectl -n kube-system get configmap kubeadm-config -o jsonpath='{.data.ClusterConfiguration}' --insecure-skip-tls-verify > kubeadm.yaml

2. Open the file in a local text editor and and find the certSANs section under apiServer. If this is not there, then it must be created. Here’s an example of what it looks like.

   apiServer:
   certSANs:
   - "10.10.10.100"
   - "kubernetes.default"
   extraArgs:
       authorization-mode: Node,RBAC
   timeoutForControlPlane: 4m0s

3. Add the new IP address or hostname to the certSANs section

   apiServer:
   certSANs:
   - "10.10.10.100"
   - "kubernetes.default"
   - "new-hostname-kubernetes"
   - "X.X.X.X"
   extraArgs:
       authorization-mode: Node,RBAC
   timeoutForControlPlane: 4m0s

4. SSH to the API server (control plane) and move the old certificates to another folder so that kubeadm can recreate new ones:

   mv /etc/kubernetes/pki/apiserver.{crt,key} ~

5. Use kubeadm to generate new apiserver certificates:

   kubeadm init phase certs apiserver --insecure-skip-tls-verify --config kubeadm.yaml

6. Now stop and restart the kubeapiserver container using docker or critools

7. Run docker ps | grep kube-apiserver | grep -v pause to get the container ID for the container running the Kubernetes API server

8. Run docker kill <containerID> to kill the container.

9. The kubelet will automatically restart the container, which will pick up the new certificates.

10. If everything is working as expected, upload the kubeadm configuration for future upgrades

    kubeadm init phase upload-config kubeadm --config kubeadm.yaml
    kubectl get nodes