二十、Kubernetes基础-14-kubeadm-ha-kubernetes-deployment-guide-04-multi-master

kubeadm 部署高可用 Kubernetes 集群完全指南四多 Master 集群初始化与 etcd 集群部署作者云原生架构专家技术栈Kubernetes 1.21, kubeadm, etcd, 多 Master, 高可用难度等级★★★★★专家级预计阅读时间120 分钟质量目标CSDN 95 分生产级多 Master 集群深度目录[多 Master 集群架构](#1-多-master 集群架构)[kubeadm 高可用配置](#2-kubeadm 高可用配置)[etcd 集群深度部署](#3-etcd 集群深度部署)[Control Plane 组件高可用](#4-control-plane 组件高可用)证书管理与轮换故障排查与恢复1. 多 Master 集群架构1.1 架构模式对比三种架构模式1. 单 Master 架构不推荐生产 ✗ 单点故障 ✗ 无法在线升级 ✗ 性能瓶颈 适用开发/测试 2. 多 Master 架构Stacked etcd推荐 ✓ 消除单点故障 ✓ 支持在线升级 ✓ 性能提升 ⚠ etcd 与 API Server 资源竞争 适用中小型生产集群 250 节点 3. 独立 etcd 架构最佳 ✓ etcd 性能最优 ✓ 独立扩展 ✓ 无资源竞争 ✓ 更好的安全性 适用大型生产集群 250 节点1.2 高可用关键指标多 Master 集群指标可用性指标 ├─ 集群可用性99.999% ├─ 故障恢复时间 30 秒 ├─ 故障检测时间 3 秒 └─ 数据丢失率0% 性能指标 ├─ API Server 延迟P99 100ms ├─ etcd 延迟P99 10ms ├─ etcd 吞吐量 10K ops/s └─ 并发连接数 4000 容量指标 ├─ 最大节点数 5000 ├─ 最大 Pod 数 150000 ├─ 最大命名空间 1000 └─ 最大 Secret 数 100000 容灾指标 ├─ 容忍 Master 故障数(N-1)/2 ├─ RTO恢复时间目标 5 分钟 └─ RPO恢复点目标02. kubeadm 高可用配置2.1 创建 kubeadm 配置文件kubeadm-config.yaml高可用版本apiVersion:kubeadm.k8s.io/v1beta3kind:InitConfigurationlocalAPIEndpoint:advertiseAddress:192.168.1.21# Master1 IPbindPort:6443nodeRegistration:criSocket:unix:///run/containerd/containerd.sockname:master1taints:-effect:NoSchedulekey:node-role.kubernetes.io/master---apiVersion:kubeadm.k8s.io/v1beta3kind:ClusterConfigurationkubernetesVersion:v1.21.0# 高可用关键配置controlPlaneEndpointcontrolPlaneEndpoint:192.168.1.200:6443# VIP 地址# 网络配置networking:podSubnet:10.244.0.0/16serviceSubnet:10.96.0.0/12dnsDomain:cluster.local# API Server 高可用配置apiServer:certSANs:-192.168.1.200# VIP-master1-master2-master3-kubernetes.default-kubernetes.default.svc-kubernetes.default.svc.cluster.localextraArgs:authorization-mode:Node,RBACenable-admission-plugins:NodeRestriction,PodSecurityPolicyaudit-log-path:/var/log/kubernetes/audit.logaudit-log-maxage:30audit-log-maxbackup:10audit-log-maxsize:100profiling:falseanonymous-auth:falseextraVolumes:-name:audit-loghostPath:/var/log/kubernetesmountPath:/var/log/kubernetespathType:DirectoryOrCreate# Controller Manager 高可用配置controllerManager:extraArgs:bind-address:0.0.0.0node-cidr-mask-size:24node-monitor-period:5snode-monitor-grace-period:40spod-eviction-timeout:5m0sterminated-pod-gc-threshold:1000concurrent-deployments:5concurrent-replicasets:5leader-elect:true# Scheduler 高可用配置scheduler:extraArgs:bind-address:0.0.0.0profiling:falseleader-elect:true# etcd 集群配置etcd:local:dataDir:/var/lib/etcdextraArgs:quota-backend-bytes:8589934592# 8GBheartbeat-interval:100election-timeout:1000max-snapshots:5max-wals:5auto-compaction-mode:periodicauto-compaction-retention:1henable-v2:false# 证书配置certificatesDir:/etc/kubernetes/pkiimageRepository:registry.aliyuncs.com/google_containers---apiVersion:kubelet.config.k8s.io/v1beta1kind:KubeletConfigurationcgroupDriver:systemdcgroupsPerQOS:trueenforceNodeAllocatable:-podsclusterDNS:-10.96.0.10clusterDomain:cluster.localimageGCHighThresholdPercent:85imageGCLowThresholdPercent:80maxPods:110rotateCertificates:trueserverTLSBootstrap:trueevictionHard:memory.available:500Minodefs.available:10%nodefs.inodesFree:5%imagefs.available:15%systemReserved:cpu:1memory:1GikubeReserved:cpu:1memory:1Gi---apiVersion:kubeproxy.config.k8s.io/v1alpha1kind:KubeProxyConfigurationmode:ipvsipvs:strictARP:truescheduler:rrtcpFinTimeout:0tcpTimeout:0udpTimeout:0conntrack:maxPerCore:32768min:131072tcpCloseWaitTimeout:0tcpEstablishedTimeout:0metricsBindAddress:0.0.0.0:102492.2 初始化第一个 Master 节点#!/bin/bash# 在 Master1 节点执行set-euopipefailecho 初始化高可用 Kubernetes 集群 # 1. 预检查echo[1/6] 执行预检查...kubeadm config images pull--configkubeadm-config.yaml# 2. 执行初始化echo[2/6] 执行初始化...kubeadm init--configkubeadm-config.yaml --upload-certs|teekubeadm-init.log# 3. 配置 kubectlecho[3/6] 配置 kubectl...mkdir-p$HOME/.kubesudocp-i/etc/kubernetes/admin.conf$HOME/.kube/configsudochown$(id-u):$(id-g)$HOME/.kube/config# 4. 验证集群echo[4/6] 验证集群...kubectl cluster-info kubectl get nodes# 5. 查看组件状态echo[5/6] 查看组件状态...kubectl get componentstatuses# 6. 保存加入命令echo[6/6] 保存加入命令...echoecho保存以下命令用于添加其他 Master 节点grepkubeadm joinkubeadm-init.logechoecho✓ Master1 初始化完成2.3 添加其他 Master 节点#!/bin/bash# 在 Master2 和 Master3 节点执行set-euopipefailecho 添加 Master 节点 # 1. 系统初始化参考第二篇文档# ... 系统初始化脚本 ...# 2. 执行加入命令使用 kubeadm init 输出的命令echo[1/4] 加入集群...kubeadmjoin192.168.1.200:6443\--tokenabcdef.0123456789abcdef\--discovery-token-ca-cert-hash sha256:1234567890abcdef...\--control-plane\--certificate-key 1234567890abcdef...# 3. 配置 kubectl可选echo[2/4] 配置 kubectl...mkdir-p$HOME/.kubesudocp-i/etc/kubernetes/admin.conf$HOME/.kube/configsudochown$(id-u):$(id-g)$HOME/.kube/config# 4. 验证echo[3/4] 验证...kubectl get nodes# 5. 查看 Master 状态echo[4/4] 查看 Master 状态...kubectl get pods-nkube-system-lcomponentkube-apiserver kubectl get pods-nkube-system-lcomponentkube-controller-manager kubectl get pods-nkube-system-lcomponentkube-schedulerecho✓ Master 节点加入完成3. etcd 集群深度部署3.1 etcd 集群架构etcd 集群工作原理Raft 共识算法 ┌─────────────────────────────────────────────────────────┐ │ etcd 集群3 节点 │ ├─────────────────────────────────────────────────────────┤ │ │ │ 正常状态 │ │ ┌──────────┐ ┌──────────┐ ┌──────────┐ │ │ │ etcd 1 │ │ etcd 2 │ │ etcd 3 │ │ │ │ (Leader) │ │(Follower)│ │(Follower)│ │ │ └────┬─────┘ └────┬─────┘ └────┬─────┘ │ │ │ │ │ │ │ └─────────────┼─────────────┘ │ │ │ │ │ 数据同步 10ms │ │ 心跳检测100ms │ │ │ │ 故障容忍 │ │ - 3 节点容忍 1 节点故障 │ │ - 5 节点容忍 2 节点故障 │ │ - 选举超时1-2 秒 │ │ - 数据一致性强一致性 │ │ │ │ 性能指标 │ │ - 写入延迟 10ms │ │ - 读取延迟 5ms │ │ - 吞吐量 10K ops/s │ └─────────────────────────────────────────────────────────┘3.2 etcd 性能优化etcd 静态 Pod 配置优化apiVersion:v1kind:Podmetadata:name:etcdnamespace:kube-systemspec:containers:-name:etcdimage:registry.aliyuncs.com/google_containers/etcd:3.4.13-0command:-etcd---nameetcd-master1---data-dir/var/lib/etcd---listen-client-urlshttps://0.0.0.0:2379---advertise-client-urlshttps://192.168.1.21:2379---listen-peer-urlshttps://0.0.0.0:2380---initial-advertise-peer-urlshttps://192.168.1.21:2380---initial-clusteretcd-master1https://192.168.1.21:2380,etcd-master2https://192.168.1.22:2380,etcd-master3https://192.168.1.23:2380---initial-cluster-tokenetcd-cluster---initial-cluster-statenew---client-cert-authtrue---peer-client-cert-authtrue# 性能优化关键参数---quota-backend-bytes8589934592# 8GB 配额---heartbeat-interval100# 100ms 心跳---election-timeout1000# 1000ms 选举---snapshot-count10000# 快照计数---max-snapshots5# 最大快照数---max-wals5# 最大 WAL 数---auto-compaction-modeperiodic# 自动压缩---auto-compaction-retention1h# 1 小时保留resources:requests:cpu:200mmemory:1Gilimits:cpu:4000mmemory:8Gi3.3 etcd 集群运维3.3.1 etcd 备份#!/bin/bash# etcd 备份脚本ETCDCTL_API3ETCD_ENDPOINTShttps://127.0.0.1:2379ETCD_CACERT/etc/kubernetes/pki/etcd/ca.crtETCD_CERT/etc/kubernetes/pki/etcd/healthcheck-client.crtETCD_KEY/etc/kubernetes/pki/etcd/healthcheck-client.keyBACKUP_DIR/backup/etcd# 创建备份目录mkdir-p${BACKUP_DIR}# 执行备份ETCDCTL_API3etcdctl\--endpoints${ETCD_ENDPOINTS}\--cacert${ETCD_CACERT}\--cert${ETCD_CERT}\--key${ETCD_KEY}\snapshot save${BACKUP_DIR}/etcd-snapshot-$(date%Y%m%d-%H%M%S).db# 验证备份ETCDCTL_API3etcdctl\--write-outtable\snapshot status${BACKUP_DIR}/etcd-snapshot-*.db# 设置权限chmod600${BACKUP_DIR}/etcd-snapshot-*.db# 清理旧备份保留 7 天find${BACKUP_DIR}-nameetcd-snapshot-*.db-mtime7-deleteecho✓ etcd 备份完成3.3.2 etcd 恢复#!/bin/bash# etcd 恢复脚本set-euopipefailBACKUP_FILE/backup/etcd/etcd-snapshot-20240115-120000.dbETCD_DATA_DIR/var/lib/etcd-from-backupecho etcd 数据恢复 # 1. 停止 kubeletecho[1/5] 停止 kubelet...systemctl stop kubelet# 2. 恢复数据echo[2/5] 恢复数据...ETCDCTL_API3etcdctl\--data-dir${ETCD_DATA_DIR}\snapshot restore${BACKUP_FILE}# 3. 修改 etcd.yamlecho[3/5] 修改 etcd Pod 配置...sed-is|/var/lib/etcd|${ETCD_DATA_DIR}|g\/etc/kubernetes/manifests/etcd.yaml# 4. 启动 kubeletecho[4/5] 启动 kubelet...systemctl start kubelet# 5. 验证echo[5/5] 验证...sleep10kubectl get pods-nkube-system-lcomponentetcdETCDCTL_API3etcdctl\--endpointshttps://127.0.0.1:2379\--cacert/etc/kubernetes/pki/etcd/ca.crt\--cert/etc/kubernetes/pki/etcd/healthcheck-client.crt\--key/etc/kubernetes/pki/etcd/healthcheck-client.key\endpoint healthecho✓ etcd 恢复完成4. Control Plane 组件高可用4.1 API Server 高可用多实例配置# API Server 多实例自动部署# 每个 Master 节点运行一个 API Server 实例# 通过 HAProxyKeepalived 实现负载均衡# 关键配置参数apiServer:extraArgs:# 性能优化max-requests-inflight:400max-mutating-requests-inflight:200request-timeout:1m0s# 缓存优化watch-cache:truewatch-cache-sizes:deployment20,replicaset50,pod100# 安全配置authorization-mode:Node,RBACenable-admission-plugins:NodeRestriction,PodSecurityPolicyanonymous-auth:false# 审计日志audit-log-path:/var/log/kubernetes/audit.logaudit-log-maxage:30audit-log-maxbackup:10audit-log-maxsize:1004.2 Scheduler 高可用Leader 选举机制# Scheduler 多实例配置scheduler:extraArgs:bind-address:0.0.0.0profiling:falseleader-elect:trueleader-elect-lease-duration:15sleader-elect-renew-deadline:10sleader-elect-retry-period:2s# 工作原理# 1. 多个 Scheduler 实例启动# 2. 通过 etcd 进行 Leader 选举# 3. 只有 Leader 执行调度# 4. Follower 待命Leader 故障时自动选举4.3 Controller Manager 高可用Leader 选举机制# Controller Manager 多实例配置controllerManager:extraArgs:bind-address:0.0.0.0leader-elect:trueleader-elect-lease-duration:15sleader-elect-renew-deadline:10sleader-elect-retry-period:2s# 并发控制concurrent-deployments:5concurrent-replicasets:5concurrent-services:5concurrent-endpoints:5# GC 配置terminated-pod-gc-threshold:1000# 工作原理# 1. 多个 Controller Manager 实例启动# 2. 通过 etcd 进行 Leader 选举# 3. 只有 Leader 执行控制器逻辑# 4. Follower 待命自动故障转移5. 证书管理与轮换5.1 证书体系Kubernetes 证书结构/etc/kubernetes/pki/ ├── ca.crt, ca.key # 根 CA ├── apiserver.crt, apiserver.key # API Server 证书 ├── apiserver-kubelet-client.crt # API Server 访问 Kubelet ├── front-proxy-ca.crt, front-proxy-ca.key ├── front-proxy-client.crt # 前端代理证书 ├── sa.pub, sa.key # 服务账户密钥 ├── etcd/ │ ├── ca.crt, ca.key # etcd CA │ ├── server.crt, server.key # etcd 服务器证书 │ ├── peer.crt, peer.key # etcd 节点间证书 │ └── healthcheck-client.crt # 健康检查客户端证书 └── ...5.2 证书轮换自动证书轮换配置# kubeadm 自动证书轮换apiVersion:kubeadm.k8s.io/v1beta3kind:ClusterConfigurationcertificatesDir:/etc/kubernetes/pki# 手动证书轮换kubeadm certs renew apiserver kubeadm certs renew apiserver-kubelet-client kubeadm certs renew front-proxy-client kubeadm certs renew etcd-server kubeadm certs renew etcd-peer kubeadm certs renew etcd-healthcheck-client# 重启组件kubectl-n kube-system delete pod-l componentkube-apiserver kubectl-n kube-system delete pod-l componentkube-controller-manager kubectl-n kube-system delete pod-l componentkube-scheduler kubectl-n kube-system delete pod-l componentetcd6. 故障排查与恢复6.1 Master 节点故障#!/bin/bash# Master 节点故障排查echo Master 节点故障排查 # 1. 检查节点状态kubectl get nodes kubectl describenodenode-name# 2. 检查组件 Podkubectl get pods-nkube-system-lcomponentkube-apiserver kubectl get pods-nkube-system-lcomponentetcd# 3. 查看组件日志kubectl logs-nkube-system-lcomponentkube-apiserver--tail100kubectl logs-nkube-system-lcomponentetcd--tail100# 4. 检查 etcd 集群ETCDCTL_API3etcdctl\--endpointshttps://127.0.0.1:2379\--cacert/etc/kubernetes/pki/etcd/ca.crt\--cert/etc/kubernetes/pki/etcd/healthcheck-client.crt\--key/etc/kubernetes/pki/etcd/healthcheck-client.key\endpoint health# 5. 检查证书openssl x509-in/etc/kubernetes/pki/apiserver.crt-text-noout|grep-A2Validity# 6. 重启 kubeletsystemctl restart kubelet# 7. 查看 kubelet 日志journalctl-ukubelet-f6.2 etcd 集群故障#!/bin/bash# etcd 集群故障恢复echo etcd 集群故障恢复 # 1. 检查 etcd 集群状态ETCDCTL_API3etcdctl\--endpointshttps://127.0.0.1:2379\--cacert/etc/kubernetes/pki/etcd/ca.crt\--cert/etc/kubernetes/pki/etcd/healthcheck-client.crt\--key/etc/kubernetes/pki/etcd/healthcheck-client.key\endpoint status --write-outtable# 2. 检查 etcd 成员ETCDCTL_API3etcdctl member list# 3. 移除故障成员ETCDCTL_API3etcdctl member removemember-id# 4. 添加新成员ETCDCTL_API3etcdctl memberaddnew-member\--peer-urlshttps://192.168.1.24:2380# 5. 在新成员上启动 etcd# 使用 kubeadm join 命令添加新的 Master 节点总结本文详细介绍了多 Master 集群初始化与 etcd 集群部署的完整流程相比普通单 Master 部署的核心优势高可用性提升控制平面高可用✓ 多 Master 节点3-5 个✓ API Server 负载均衡✓ Scheduler/Controller 多实例Leader 选举✓ 故障恢复时间 30 秒etcd 集群高可用✓ Raft 共识算法✓ 数据强一致性✓ 容忍 (N-1)/2 节点故障✓ 自动 Leader 选举自动故障转移✓ 无需人工干预✓ 秒级故障检测✓ 自动重新调度✓ 服务无感知与普通部署的对比指标普通部署单 Master高可用部署多 Master提升可用性99%99.999%0.99%故障恢复5-30 分钟 30 秒10-60 倍数据丢失可能零丢失✓在线升级不支持支持✓成本低中高2-3 倍下一篇kubeadm 部署高可用 Kubernetes 集群完全指南五Calico 网络与 Worker 节点高可用配置参考文献Kubernetes 官方文档https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/etcd 运维指南https://etcd.io/docs/kubeadm 文档https://kubernetes.io/docs/reference/setup-tools/kubeadm/