在国产麒麟V10系统上，用kubeadm一步步搭建3个master节点的k8s高可用集群（含haproxy+keepalived配置）

国产麒麟V10系统上构建高可用Kubernetes集群实战指南在信息技术自主可控的大背景下国产操作系统正逐步成为企业级基础设施的重要选择。本文将详细介绍如何在麒麟V10Kylin V10操作系统上从零开始搭建一个包含3个Master节点的高可用Kubernetes集群。不同于通用Linux发行版的部署方案我们将特别关注国产化环境中的特殊配置和优化技巧帮助您构建一个稳定、高效且符合企业级标准的容器化平台。1. 环境准备与系统调优麒麟V10作为国产操作系统的代表其底层基于Linux内核进行了深度定制和优化。在部署Kubernetes集群前我们需要对系统进行一系列针对性配置确保其满足容器编排平台的基础要求。1.1 系统基础配置首先我们需要为每个节点配置唯一的主机名和静态IP地址。在麒麟V10中可以通过以下命令修改主机名hostnamectl set-hostname master1 # 在master1节点执行 hostnamectl set-hostname master2 # 在master2节点执行 hostnamectl set-hostname master3 # 在master3节点执行接下来编辑/etc/hosts文件确保所有节点能够通过主机名互相解析192.168.1.1 master1 192.168.1.2 master2 192.168.1.3 master3 192.168.1.11 node1 192.168.1.12 node21.2 内核参数优化Kubernetes对Linux内核参数有特定要求我们需要调整以下关键参数cat /etc/sysctl.d/k8s.conf EOF net.bridge.bridge-nf-call-ip6tables 1 net.bridge.bridge-nf-call-iptables 1 net.ipv4.ip_forward 1 vm.swappiness 0 EOF sysctl -p /etc/sysctl.d/k8s.conf然后加载必要的内核模块modprobe br_netfilter modprobe overlay lsmod | grep -e br_netfilter -e overlay1.3 系统服务配置为提高部署成功率建议暂时关闭防火墙和SELinuxsystemctl stop firewalld systemctl disable firewalld setenforce 0 sed -i s/SELINUXenforcing/SELINUXdisabled/g /etc/selinux/config同时关闭Swap分区swapoff -a sed -i /swap/s/^/#/ /etc/fstab注意在生产环境中应根据实际安全需求评估是否重新启用防火墙和SELinux并配置相应的放行规则。2. 容器运行时与Kubernetes组件安装2.1 容器运行时选择与配置在麒麟V10上我们推荐使用containerd作为容器运行时。首先安装必要依赖yum install -y libseccomp-devel socat conntrack然后下载并安装containerdwget https://github.com/containerd/containerd/releases/download/v1.7.2/containerd-1.7.2-linux-amd64.tar.gz tar Cxzvf /usr/local containerd-1.7.2-linux-amd64.tar.gz配置systemd服务wget https://raw.githubusercontent.com/containerd/containerd/main/containerd.service mkdir -p /usr/local/lib/systemd/system mv containerd.service /usr/local/lib/systemd/system/ systemctl daemon-reload systemctl enable --now containerd生成默认配置文件并启用systemd cgroup驱动mkdir -p /etc/containerd containerd config default /etc/containerd/config.toml sed -i s/SystemdCgroup false/SystemdCgroup true/g /etc/containerd/config.toml systemctl restart containerd2.2 Kubernetes组件安装配置Kubernetes的YUM源使用国内镜像加速cat /etc/yum.repos.d/kubernetes.repo EOF [kubernetes] nameKubernetes baseurlhttps://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled1 gpgcheck1 repo_gpgcheck1 gpgkeyhttps://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF安装kubelet、kubeadm和kubectlyum install -y kubelet-1.28.2 kubeadm-1.28.2 kubectl-1.28.2 --disableexcludeskubernetes systemctl enable kubelet3. 高可用架构部署3.1 HAProxy负载均衡配置在所有Master节点上安装HAProxyyum install -y haproxy配置HAProxy/etc/haproxy/haproxy.cfgcat /etc/haproxy/haproxy.cfg EOF global log /dev/log local0 log /dev/log local1 notice chroot /var/lib/haproxy stats socket /run/haproxy/admin.sock mode 660 level admin stats timeout 30s user haproxy group haproxy daemon defaults log global mode tcp option tcplog option dontlognull timeout connect 5000 timeout client 50000 timeout server 50000 frontend k8s-api bind *:16443 default_backend k8s-api-servers backend k8s-api-servers balance roundrobin server master1 192.168.1.1:6443 check server master2 192.168.1.2:6443 check server master3 192.168.1.3:6443 check EOF启动并验证HAProxy服务systemctl enable haproxy systemctl restart haproxy ss -lntp | grep 164433.2 Keepalived虚拟IP管理安装Keepalivedyum install -y keepalived配置主节点master1的Keepalivedcat /etc/keepalived/keepalived.conf EOF ! Configuration File for keepalived global_defs { router_id LVS_MASTER } vrrp_script chk_haproxy { script killall -0 haproxy interval 2 weight 2 } vrrp_instance VI_1 { state MASTER interface eth0 virtual_router_id 51 priority 100 advert_int 1 authentication { auth_type PASS auth_pass 42 } virtual_ipaddress { 192.168.1.10/24 } track_script { chk_haproxy } } EOF在备份节点master2和master3上只需修改state为BACKUP并降低priority值如90和80。启动服务systemctl enable keepalived systemctl restart keepalived ip addr show eth0 | grep 192.168.1.104. Kubernetes集群初始化与扩展4.1 主节点初始化配置创建kubeadm配置文件kubeadm-config.yamlapiVersion: kubeadm.k8s.io/v1beta3 kind: ClusterConfiguration kubernetesVersion: 1.28.2 controlPlaneEndpoint: 192.168.1.10:16443 imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers networking: podSubnet: 10.244.0.0/16 serviceSubnet: 10.96.0.0/12 apiServer: certSANs: - 192.168.1.1 - 192.168.1.2 - 192.168.1.3 - 192.168.1.10 --- apiVersion: kubeadm.k8s.io/v1beta3 kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.1.1 nodeRegistration: criSocket: unix:///run/containerd/containerd.sock kubeletExtraArgs: cgroup-driver: systemd初始化第一个Master节点kubeadm init --configkubeadm-config.yaml --upload-certs配置kubectlmkdir -p $HOME/.kube cp -i /etc/kubernetes/admin.conf $HOME/.kube/config chown $(id -u):$(id -g) $HOME/.kube/config4.2 添加额外控制平面节点将证书文件从master1复制到其他Master节点for node in master2 master3; do ssh $node mkdir -p /etc/kubernetes/pki/etcd scp /etc/kubernetes/pki/ca.* $node:/etc/kubernetes/pki/ scp /etc/kubernetes/pki/sa.* $node:/etc/kubernetes/pki/ scp /etc/kubernetes/pki/front-proxy-ca.* $node:/etc/kubernetes/pki/ scp /etc/kubernetes/pki/etcd/ca.* $node:/etc/kubernetes/pki/etcd/ done在其他Master节点上执行加入命令命令来自master1初始化输出kubeadm join 192.168.1.10:16443 --token token \ --discovery-token-ca-cert-hash sha256:hash \ --control-plane --certificate-key cert-key4.3 添加Worker节点在Worker节点上执行加入命令kubeadm join 192.168.1.10:16443 --token token \ --discovery-token-ca-cert-hash sha256:hash4.4 网络插件安装部署Calico网络插件适配我们配置的podSubnetkubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml验证集群状态kubectl get nodes -o wide kubectl get pods -n kube-system -o wide5. 集群验证与优化5.1 高可用性测试验证虚拟IP的故障转移能力在master1上停止keepalived服务systemctl stop keepalived观察虚拟IP是否迁移到master2或master3检查Kubernetes API是否仍然可用kubectl get nodes恢复master1的keepalived服务systemctl start keepalived5.2 性能调优建议调整kubelet配置/var/lib/kubelet/config.yamlapiVersion: kubelet.config.k8s.io/v1beta1 kind: KubeletConfiguration evictionHard: memory.available: 500Mi nodefs.available: 10% nodefs.inodesFree: 5% imagefs.available: 15% kubeReserved: cpu: 500m memory: 500Mi systemReserved: cpu: 500m memory: 500Mi重启kubelet使配置生效systemctl restart kubelet5.3 关键组件监控部署Prometheus监控栈kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/setup.yaml kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/kube-prometheus/main/manifests/配置HAProxy和Keepalived的监控cat /etc/prometheus/haproxy.yaml EOF - job_name: haproxy static_configs: - targets: [localhost:8404] EOF