k8s apiserver高可用方案

目前官方推荐有 2 种方式部署k8s apiserver 高可用

keepalived and haproxy

部署有2种方式，一种是systemd管理的，另一种是pod形式，使用那种可以根据实际情况选择

服务部署

systemd方式

可以通过包管理工具安装，正常启动之后，通过 kubeadm init 安装集群，配置好 apiserver 地址 VIP:${APISERVER_DEST_PORT}，配置参数见下面👇

pod方式

如果 keepalived 和 haproxy 将在控制平面节点上运行，则可以将它们配置作为静态 Pod 运行，这里所需要的只是在引导集群之前将相应的清单文件放置在 /etc/kubernetes/manifests 目录中，在引导过程中，kubelet 将启动这些 pod，这种方式比较优雅

keepalived

apiVersion: v1
kind: Pod
metadata:creationTimestamp: nullname: keepalivednamespace: kube-system
spec:containers:- image: osixia/keepalived:2.0.20name: keepalivedresources: {}securityContext:capabilities:add:- NET_ADMIN- NET_BROADCAST- NET_RAWvolumeMounts:- mountPath: /usr/local/etc/keepalived/keepalived.confname: config- mountPath: /etc/keepalived/check_apiserver.shname: checkhostNetwork: truevolumes:- hostPath:path: /etc/keepalived/keepalived.confname: config- hostPath:path: /etc/keepalived/check_apiserver.shname: check
status: {}

apiVersion: v1
kind: Pod
metadata:name: haproxynamespace: kube-system
spec:containers:- image: haproxy:2.8name: haproxylivenessProbe:failureThreshold: 8httpGet:host: localhostpath: /healthzport: ${APISERVER_DEST_PORT}scheme: HTTPSvolumeMounts:- mountPath: /usr/local/etc/haproxy/haproxy.cfgname: haproxyconfreadOnly: truehostNetwork: truevolumes:- hostPath:path: /etc/haproxy/haproxy.cfgtype: FileOrCreatename: haproxyconf
status: {}

${APISERVER_DEST_PORT} 值需要和配置文件一致，配置好后，执行kubeadm init引导集群

服务配置

keepalived

keepalived配置包含服务配置文件和健康检查文件，配置文件放在**/etc/keepalived**目录，以下配置文件适用于2.0.20 and 2.2.4

! /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {router_id LVS_DEVEL
}
vrrp_script check_apiserver {script "/etc/keepalived/check_apiserver.sh"interval 3weight -2fall 10rise 2
}vrrp_instance VI_1 {state ${STATE}interface ${INTERFACE}virtual_router_id ${ROUTER_ID}priority ${PRIORITY}authentication {auth_type PASSauth_pass ${AUTH_PASS}}virtual_ipaddress {${APISERVER_VIP}}track_script {check_apiserver}
}

根据具体情况把上面配置文件变量替换成具体值

• ${STATE} 对于一台主机来说是 MASTER，对于所有其他主机来说是 BACKUP，虚拟 IP 最初将分配给 MASTER。
• ${INTERFACE} VIP 需要绑定的网络接口，例如eth0。
• ${ROUTER_ID} 应该相同，但在同一子网中的所有集群中是唯一的。许多发行版将其值预先配置为 51，需要确认清楚
• ${PRIORITY} MASTER 节点应高于BACKUP节点上的 ${PRIORITY}。因此 101 和 100 分别就足够了。
  对于所有 keepalived 集群主机，
• ${AUTH_PASS} 验证密码，所有配置文件保持一致，例如123456
• ${APISERVER_VIP} 需要使用的 VIP 地址。

keepalived的健康检查脚本放在/etc/keepalived/check_apiserver.sh。

#!/bin/sh
APISERVER_DEST_PORT=6443
errorExit() {echo "*** $*" 1>&2exit 1
}curl -sfk --max-time 2 https://localhost:${APISERVER_DEST_PORT}/healthz -o /dev/null || errorExit "Error GET https://localhost:${APISERVER_DEST_PORT}/healthz"

haproxy

配置文件目录/etc/haproxy，适配版本 2.4 and 2.8

# /etc/haproxy/haproxy.cfg
#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
globallog stdout format raw local0daemon#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------
defaultsmode                    httplog                     globaloption                  httplogoption                  dontlognulloption http-server-closeoption forwardfor       except 127.0.0.0/8option                  redispatchretries                 1timeout http-request    10stimeout queue           20stimeout connect         5stimeout client          35stimeout server          35stimeout http-keep-alive 10stimeout check           10s#---------------------------------------------------------------------
# apiserver frontend which proxys to the control plane nodes
#---------------------------------------------------------------------
frontend apiserverbind *:${APISERVER_DEST_PORT}mode tcpoption tcplogdefault_backend apiserverbackend#---------------------------------------------------------------------
# round robin balancing for apiserver
#---------------------------------------------------------------------
backend apiserverbackendoption httpchkhttp-check connect sslhttp-check send meth GET uri /healthzhttp-check expect status 200mode tcpbalance     roundrobinserver ${HOST1_ID} ${HOST1_ADDRESS}:${APISERVER_SRC_PORT} check verify none# [...]

• ${APISERVER_DEST_PORT} haproxy 监听的端口，转发 apiserver 请求
• ${APISERVER_SRC_PORT} API Server 实例使用的端口
• ${HOST1_ID} 第一个负载平衡 API Server 主机名称，随意起，有多个可以再添加多个 server
• ${HOST1_ADDRESS} 第一个负载平衡 API Server 主机的可解析地址（DNS 名称、IP 地址）

kube-vip

kube-vip官方文档