Kubernetes: expired certificate
I think you need re-generate the apiserver certificate /etc/kubernetes/pki/apiserver.crt
you can view current expire date like this.
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
Not Before: Dec 20 14:32:00 2017 GMT
Not After : Dec 20 14:32:00 2018 GMT
Here is the steps I used to regenerate the certificates on v1.11.5 cluster. compiled steps from here https://github.com/kubernetes/kubeadm/issues/581
to check all certificate expire date:
find /etc/kubernetes/pki/ -type f -name "*.crt" -print|egrep -v 'ca.crt$'|xargs -L 1 -t -i bash -c 'openssl x509 -noout -text -in {}|grep After'
Renew certificate on Master node.
*) Renew certificate
mv /etc/kubernetes/pki/apiserver.key /etc/kubernetes/pki/apiserver.key.old
mv /etc/kubernetes/pki/apiserver.crt /etc/kubernetes/pki/apiserver.crt.old
mv /etc/kubernetes/pki/apiserver-kubelet-client.crt /etc/kubernetes/pki/apiserver-kubelet-client.crt.old
mv /etc/kubernetes/pki/apiserver-kubelet-client.key /etc/kubernetes/pki/apiserver-kubelet-client.key.old
mv /etc/kubernetes/pki/front-proxy-client.crt /etc/kubernetes/pki/front-proxy-client.crt.old
mv /etc/kubernetes/pki/front-proxy-client.key /etc/kubernetes/pki/front-proxy-client.key.old
kubeadm alpha phase certs apiserver --config /root/kubeadm-kubetest.yaml
kubeadm alpha phase certs apiserver-kubelet-client
kubeadm alpha phase certs front-proxy-client
mv /etc/kubernetes/pki/apiserver-etcd-client.crt /etc/kubernetes/pki/apiserver-etcd-client.crt.old
mv /etc/kubernetes/pki/apiserver-etcd-client.key /etc/kubernetes/pki/apiserver-etcd-client.key.old
kubeadm alpha phase certs apiserver-etcd-client
mv /etc/kubernetes/pki/etcd/server.crt /etc/kubernetes/pki/etcd/server.crt.old
mv /etc/kubernetes/pki/etcd/server.key /etc/kubernetes/pki/etcd/server.key.old
kubeadm alpha phase certs etcd-server --config /root/kubeadm-kubetest.yaml
mv /etc/kubernetes/pki/etcd/healthcheck-client.crt /etc/kubernetes/pki/etcd/healthcheck-client.crt.old
mv /etc/kubernetes/pki/etcd/healthcheck-client.key /etc/kubernetes/pki/etcd/healthcheck-client.key.old
kubeadm alpha phase certs etcd-healthcheck-client --config /root/kubeadm-kubetest.yaml
mv /etc/kubernetes/pki/etcd/peer.crt /etc/kubernetes/pki/etcd/peer.crt.old
mv /etc/kubernetes/pki/etcd/peer.key /etc/kubernetes/pki/etcd/peer.key.old
kubeadm alpha phase certs etcd-peer --config /root/kubeadm-kubetest.yaml
*) Backup old configuration files
mv /etc/kubernetes/admin.conf /etc/kubernetes/admin.conf.old
mv /etc/kubernetes/kubelet.conf /etc/kubernetes/kubelet.conf.old
mv /etc/kubernetes/controller-manager.conf /etc/kubernetes/controller-manager.conf.old
mv /etc/kubernetes/scheduler.conf /etc/kubernetes/scheduler.conf.old
kubeadm alpha phase kubeconfig all --config /root/kubeadm-kubetest.yaml
mv $HOME/.kube/config .$HOMEkube/config.old
cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config
chmod 777 $HOME/.kube/config
export KUBECONFIG=.kube/config
Reboot the node and check the logs for etcd, kubeapi and kubelet.
Note: Remember to update your CI/CD job kubeconfig file. If you’re using helm command test that also.
Each node within the Kubernetes cluster contains a config file for running kubelet ... /etc/kubernetes/kubelet.conf
... and this file is auto-generated by kubeadm. During this auto-generation, kubeadm uses /etc/kubernetes/ca.key
to create a node-specific file, /etc/kubernetes/kubelet.conf
, within which are two very important pieces ... client-certificate-data and client-key-data. My original thought process led me to believe that I needed to find the corresponding certificate file & key file, renew those files, convert both to base64, and use those values within kubelet.conf
files across the cluster ... this thinking was not correct.
Instead, the fix was to use kubeadm to regenerate kubectl.conf
on all nodes, as well as admin.conf
, controller-manager.conf
, and scheduler.conf
on the cluster's master node. You'll need /etc/kubernetes/pki/ca.key
on each node in order for your config files to include valid data for client-certificate-data and client-key-data.
Pro tip: make use of the --apiserver-advertise-address
parameter to ensure your new config files contain the correct IP address of the node hosting the kube-apiserver service.
This topic is also discussed in:
- https://github.com/kubernetes/kubeadm/issues/581
- after 1.15 kubeadm upgrade automatically will renewal the certificates for you!
- also 1.15 added a command to check cert expiration in kubeadm
- Renew kubernetes pki after expired
Kubernetes v1.15 provides docs for "Certificate Management with kubeadm":
- https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-certs/
- Check certificate expiration:
kubeadm alpha certs check-expiration
- Automatic certificate renewal:
- kubeadm renews all the certificates during control plane upgrade.
- Manual certificate renewal:
- You can renew your certificates manually at any time with the
kubeadm alpha certs renew
command. - This command performs the renewal using CA (or front-proxy-CA) certificate and key stored in /etc/kubernetes/pki.
- You can renew your certificates manually at any time with the
For Kubernetes v1.14 I find this procedure the most helpful:
- https://stackoverflow.com/a/56334732/1147487
- backup and re-generate all certs:
$ cd /etc/kubernetes/pki/
$ mv {apiserver.crt,apiserver-etcd-client.key,apiserver-kubelet-client.crt,front-proxy-ca.crt,front-proxy-client.crt,front-proxy-client.key,front-proxy-ca.key,apiserver-kubelet-client.key,apiserver.key,apiserver-etcd-client.crt} ~/
$ kubeadm init phase certs all --apiserver-advertise-address <IP>
- backup and re-generate all kubeconfig files:
$ cd /etc/kubernetes/
$ mv {admin.conf,controller-manager.conf,kubelet.conf,scheduler.conf} ~/
$ kubeadm init phase kubeconfig all
$ reboot
- copy new admin.conf:
$ cp -i /etc/kubernetes/admin.conf $HOME/.kube/config