k8s集群master故障恢复笔记

剔除故障节点

kubectl drain master故障节点

kubectl delete node master故障节点

kubeadm reset

rm -rf /etc/kubernetes/manifests

mkdir -p /etc/kubernetes/pki/etcd/

从master其他节点拷

scp /etc/kubernetes/pki/ca.crt ca.key sa.key sa.pub front-proxy-ca.crt front-proxy-ca.key master故障节点:/etc/kubernetes/pki/

scp /etc/kubernetes/admin.conf master故障节点:/etc/kubernetes/

scp /etc/kubernetes/pki/etcd/ca.crt ca.key master故障节点:/etc/kubernetes/pki/etcd/

删除etcd信息

在master01节点上执行命令，进入etcd的容器里

kubectl exec -it etcd-master01 sh

etcdctl --endpoints 127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member list

因为我们的master03机器对应的hash是xxxx8ecc13xxxx。我们下一步就是根据hash删除etcd信息，执行如下命令

etcdctl --endpoints 127.0.0.1:2379 --cacert /etc/kubernetes/pki/etcd/ca.crt --cert /etc/kubernetes/pki/etcd/server.crt --key /etc/kubernetes/pki/etcd/server.key member remove xxxx8ecc13xxxx

获取添加master的命令

kubeadm init phase upload-certs --upload-certs

xxxxxxxbxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

kubeadm token create --print-join-command

通过组合上面的“蓝色字体部分“+“–control-plane –certificate-key“ +“红色字体部分”，获得加入master的完整命令

kubeadm join xxxxxxxxx:6443 --token vc713d.vftzjn61i32frhbe --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxxxxxxxxxx

故障节点重新加入集群

kubeadm join xxxxxxxxxxx:6443 --token vc713d.vftzjn61i32frhbe --discovery-token-ca-cert-hash sha256:xxxxxxxxxxxxxxx --control-plane –certificate-key xxxxxxxxxxxxxxxxxxxxxxxxxx