高可用集群keepalived 原理+实战

1.高可用集群

1.1简介

Keepalived是Linux下一个轻量级别的高可用解决方案。高可用：广义来讲，是指整个系统的高可用行；狭义的来讲就是主机的冗余和接管。

Keepalived起初是为LVS设计的，专门用来监控集群系统中各个服务节点的状态，它根据TCP/IP参考模型的第三、第四层、第五层交换机制检测每个服务节点的状态，如果某个服务器节点出现异常，或者工作出现故障，Keepalived将检测到，并将出现的故障的服务器节点从集群系统中剔除，这些工作全部是自动完成的，不需要人工干涉，需要人工完成的只是修复出现故障的服务节点。

后来Keepalived又加入了VRRP的功能，VRRP（VritrualRouterRedundancyProtocol,虚拟路由冗余协议)出现的目的是解决静态路由出现的单点故障问题，通过VRRP可以实现网络不间断稳定运行，因此Keepalvied一方面具有服务器状态检测和故障隔离功能，另外一方面也有HAcluster功能。

健康检查和失败切换是keepalived的两大核心功能。所谓的健康检查，就是采用tcp三次握手，icmp请求，http请求，udp
echo请求等方式对负载均衡器后面的实际的服务器(通常是承载真实业务的服务器)进行保活；而失败切换主要是应用于配置了主备模式的负载均衡器，利用VRRP维持主备负载均衡器的心跳，当主负载均衡器出现问题时，由备负载均衡器承载对应的业务，从而在最大限度上减少流量损失，并提供服务的稳定性。

1.2原理

Keepalived对服务器运行状态和故障隔离的工作原理：

Keepalived工作在TCP/IP参考模型的三层、四层、五层（物理层，链路层）：

• 网络层（3）：Keepalived通过ICMP协议向服务器集群中的每一个节点发送一个ICMP数据包(有点类似与Ping的功能)，如果某个节点没有返回响应数据包，那么认为该节点发生了故障，Keepalived将报告这个节点失效，并从服务器集群中剔除故障节点。

• 传输层（4）：Keepalived在传输层里利用了TCP协议的端口连接和扫描技术来判断集群节点的端口是否正常，比如对于常见的WEB服务器80端口。或者SSH服务22端口，Keepalived一旦在传输层探测到这些端口号没有数据响应和数据返回，就认为这些端口发生异常，然后强制将这些端口所对应的节点从服务器集群中剔除掉。

• 应用层（5）：，Keepalived的运行方式也更加全面化和复杂化，用户可以通过自定义Keepalived工作方式，例如：可以通过编写程序或者脚本来运行Keepalived，而Keepalived将根据用户的设定参数检测各种程序或者服务是否允许正常，如果Keepalived的检测结果和用户设定的不一致时，Keepalived将把对应的服务器从服务器集群中剔除。

1.3 集群类型

• LB：Load Balance 负载均衡
  LVS/HAProxy/nginx（http/upstream, stream/upstream）
• HA：High Availability 高可用集群
  数据库、Redis
• SPoF: Single Point of Failure，解决单点故障
  HPC：High Performance Computing 高性能集群

1.4实现高可用

提升系统高用性的解决方案：降低MTTR- Mean Time To Repair(平均故障时间)
解决方案：建立冗余机制

• active/passive 主/备
• active/active 双主
• active --> HEARTBEAT --> passive
• active <–> HEARTBEAT <–> active

1.5VRRP：Virtual Router Redundancy Protocol

虚拟路由冗余协议,解决静态网关单点风险
物理层:路由器、三层交换机
软件层:keepalived

1.5.1 VRRP 相关术语

虚拟路由器：Virtual Router
虚拟路由器标识：VRID(0-255)，唯一标识虚拟路由器
VIP：Virtual IP
VMAC：Virutal MAC (00-00-5e-00-01-VRID)
物理路由器：
master：主设备
backup：备用设备
priority：优先级

1.5.2VRRP 相关技术

通告：心跳，优先级等；周期性
工作方式：抢占式，非抢占式
安全认证：
无认证
简单字符认证：预共享密钥
MD5
工作模式：
主/备：单虚拟路由器
主/主：主/备（虚拟路由器1），备/主（虚拟路由器2）

2.实验

2.1keepalived环境部署

功能	ip
KA1	172.25.254.10
KA2	172.25.254.20
realserver1	172.25.254.110
realserver2	172.252.54.120

#realserver1、realserver2
yum install httpd -y
​
[root@realserver1 ~]# echo realserver1 - 172.25.254.110 > /var/www/html/index.html
[root@realserver1 ~]# systemctl enable --now httpd
[root@realserver2 ~]# echo realserver2 - 172.25.254.120 > /var/www/html/index.html
[root@realserver2 ~]# systemctl enable --now httpd

#ka1、ka2访问
curl 172.25.254.110
realserver1
curl 172.25.254.120
realserver2

虚拟路由管理

#ka1、ka2
[root@ka1 ~]# yum install keepalived -y
[root@ka2 ~]# yum install keepalived -y

#全局配置
[root@ka1 ~]# vim /etc/keepalived/keepalived.conf
global_defs {notification_email {[e-mail]}notification_email_from [e-mail]_server 127.0.0.1smtp_connect_timeout 30router_id ka1.timinglee.orgvrrp_skip_check_adv_addrvrrp_strictvrrp_garp_interval 0vrrp_gna_interval 0vrrp_mcast_group4 224.0.0.18
}

#虚拟路由器配置
vrrp_instance VI_1 {state MASTERinterface eth0virtual_router_id 100priority 100advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.100/24 dev eth0 label eth0:1}
}

[root@ka1 ~]# systemctl enable --now keepalived.service
[root@ka1 ~]# ifconfig
[root@ka1 ~]# scp /etc/keepalived/keepalived.conf root@172.25.254.20:/etc/keepalived/keepalived.conf 
# 在ka2上修改这两条，先不重新启动，查看是那个被用
vrrp_instance VI_1 {state BACKUPpriority 80

#测试
[root@ka1 ~]# tcpdump -i eth0 -nn host 224.0.0.18
23:04:18.883958 IP 172.25.254.10 > 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 100, authtype simple, intvl 1s, length 20
#关掉ka1的keepalived，会出现另一台的
23:04:05.872730 IP 172.25.254.20 > 224.0.0.18: VRRPv2, Advertisement, vrid 100, prio 80, authtype simple, intvl 1s, length 20

启用keepalived日志功能

[root@ka1 ~]# vim /etc/sysconfig/keepalived 
KEEPALIVED_OPTIONS="-D -S 6"
[root@ka1 ~]# vim /etc/rsyslog.conf 
local6.*                                                /var/log/keepa
lived.log
[root@ka1 ~]# systemctl restart keepalived.service 
[root@ka1 ~]# systemctl restart rsyslog.service 
[root@ka1 ~]# ll /var/log/keepalived.log 

独立子配置文件

[root@ka1 ~]# vim /etc/keepalived/keepalived.conf 
#vrrp_instance VI_1 {
#    state MASTER
#    interface eth0
#    virtual_router_id 100
#    priority 100
#    advert_int 1
#    authentication {
#        auth_type PASS
#        auth_pass 1111
#    }
#    virtual_ipaddress {
#        172.25.254.100/24 dev eth0 label eth0:1
#    }
#}
include "/etc/keepalived/conf.d/*.conf"

[root@ka1 ~]# mkdir -p /etc/keepalived/conf.d
[root@ka1 ~]# vim /etc/keepalived/conf.d/172.25.254.100.conf
vrrp_instance VI_1 {state MASTERinterface eth0virtual_router_id 100priority 100advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.100/24 dev eth0 label eth0:1}
}
​
[root@ka1 ~]# systemctl restart keepalived.service 
[root@ka1 ~]# ifconfig

2.2抢占模式和非抢占模式

2.2.1非抢占模式

非抢占模式 nopreempt
默认为抢占模式preempt，即当高优先级的主机恢复在线后，会抢占低先级的主机的master角色，
这样会使vip在KA主机中来回漂移，造成网络抖动，
建议设置为非抢占模式 nopreempt ，即高优先级主机恢复后，并不会抢占低优先级主机的master角色
非抢占模块下,如果原主机down机, VIP迁移至的新主机, 后续也发生down时,仍会将VIP迁移回原主机

注意：要关闭 VIP抢占，必须将各 keepalived 服务器state配置为BACKUP

#ka1主机配置
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 20
priority 100 #优先级高
nopreempt #非抢占模式
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.100/24 dev eth0 label eth0:0
}
}

#KA2主机配置
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 20
priority 80 #优先级低
advert_int 1
nopreempt #非抢占模式
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.100/24 dev eth0 label eth0:0
}
}

2.2.2抢占延迟模式 preempt_delay

抢占延迟模式，即优先级高的主机恢复后，不会立即抢回VIP，而是延迟一段时间（默认300s）再抢回VIP

 preempt_delay # #指定抢占延迟时间为#s，默认延迟300s

注意：需要各keepalived服务器state为BACKUP,并且不要启用 vrrp_strict

#ka1主机配置
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 20
priority 100 #优先级高
preempt_delay 10s #抢占延迟10s
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.100/24 dev eth0 label eth0:0
}
}

#KA2主机配置
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 20
priority 80 #优先级低
advert_int 1
preempt_delay 10s #抢占延迟10S
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.100/24 dev eth0 label eth0:0
}
}

2.3VIP的单播配置

默认keepalived主机之间利用多播相互通告消息，会造成网络拥塞，可以替换成单播，减少网络流量
#在所有节点vrrp_instance语句块中设置对方主机的IP，建议设置为专用于对应心跳线网络的地址，而非使用业务网络

master主机配置

[root@KA1 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
[e-mail]
}
notification_email_from [e-mail]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id KA1.timinglee.org
vrrp_skip_check_adv_addr
#vrrp_strict #注释此参数，与vip单播模式冲突
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_ipsets keepalived
}
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 20
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.100/24 dev eth0 label eth0:0
}
unicast_src_ip 172.25.254.20 #本机IP
unicast_peer {
172.25.254.30 #指向对方主机IP
#如果有多个keepalived,再加其它节点的IP
}
}

salve主机配置

[root@KA2 ~]# vim /etc/keepalived/keepalived.conf
! Configuration File for keepalived
global_defs {
notification_email {
[e-mail]
}
notification_email_from [e-mail]
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id KA1.timinglee.org
vrrp_skip_check_adv_addr
#vrrp_strict #注释此参数，与vip单播模式冲突
vrrp_garp_interval 0
vrrp_gna_interval 0
vrrp_ipsets keepalived
}
vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 20
priority 80
advert_int 1
preempt_delay 60
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.100/24 dev eth0 label eth0:0
}
unicast_src_ip 172.25.254.30 #本机ip
unicast_peer {
172.25.254.20 #对端主机IP
}
}

测试：抓包查看单播效果

[root@KA1 ~]# tcpdump -i eth0 -nn src host 172.25.254.20 and dst 172.25.254.30
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
00:20:16.150917 IP 172.25.254.10 > 172.25.254.20: VRRPv2, Advertisement, vrid 20,
prio 100, authtype simple, intvl 1s, length 20
00:20:17.151569 IP 172.25.254.10 > 172.25.254.20: VRRPv2, Advertisement, vrid 20,
prio 100, authtype simple, intvl 1s, length 20
00:20:18.151754 IP 172.25.254.10 > 172.25.254.20: VRRPv2, Advertisement, vrid 20,
prio 100, authtype simple, intvl 1s, length 20
00:20:19.152290 IP 172.25.254.10 > 172.25.254.20: VRRPv2, Advertisement, vrid 20,
prio 100, authtype simple, intvl 1s, length 20[root@KA2 ~]# tcpdump -i eth0 -nn src 172.25.254.20 and dst 172.25.254.10
dropped privs to tcpdump
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
00:20:50.853174 IP 172.25.254.20 > 172.25.254.10: VRRPv2, Advertisement, vrid 30,
prio 80, authtype simple, intvl 1s, length 20
00:20:51.853798 IP 172.25.254.20 > 172.25.254.10: VRRPv2, Advertisement, vrid 30,
prio 80, authtype simple, intvl 1s, length 20

2.4Keepalived 通知脚本配置

当keepalived的状态变化时，可以自动触发脚本的执行，比如：发邮件通知用户
默认以用户keepalived_script身份执行脚本
如果此用户不存在，以root执行脚本可以用下面指令指定脚本执行用户的身份

2.4.1邮件配置

# 安装邮件发送工具
[root@KA1 ~]# yum install mailx -y

[root@KA1 ~]# vim /etc/mail.rc
#######mail set##########
set from=[e-mail]
set smtp=smtp.qq.com
set smtp-auth-user=[e-mail]
set smtp-auth-password=angelababybeau   #自己邮箱的授权码
set smtp-auth=login
set ssl-verify=ignore
​
#发送邮件
echo hello word | mail -s test [e-mail]

[root@ka1 ~]# vim /etc/keepalived/mail.sh 

 chmod +x /etc/keepalived/mail.sh

[root@KA1 +K2 ~]#vim /etc/keepalived/keepalived.conf
vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 20
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
172.25.254.100/24 dev eth0 label eth0:0
}
unicast_src_ip 172.25.254.20
unicast_peer {
172.25.254.30
}
notify_master "/etc/keepalived/mail.sh master"
notify_backup "/etc/keepalived/mail.sh backup"
notify_fault "/etc/keepalived/mail.sh fault"
}

#模拟master故障
[root@ka1-centos8 ~]#killall keepalived

测试

2.5实现 master/master 的 Keepalived 双主架构

master/slave的单主架构，同一时间只有一个Keepalived对外提供服务，此主机繁忙，而另一台主机却很空闲，利用率低下，可以使用master/master的双主架构，解决此问题。

master/master 的双主架构：
即将两个或以上VIP分别运行在不同的keepalived服务器，以实现服务器并行提供web访问的目的，提高服务器资源利用率

# ka1配置
vrrp_instance VI_1 {state MASTERinterface eth0virtual_router_id 100priority 100advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.100/24 dev eth0 label eth0:1}unicast_src_ip 172.25.254.10unicast_peer {172.25.254.20}track_script {check_haproxy}
}
vrrp_instance VI_2 {state BACKUPinterface eth0virtual_router_id 200priority 80advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.200/24 dev eth0 label eth0:2}unicast_src_ip 172.25.254.10unicast_peer {172.25.254.20}
}

# ka2配置
vrrp_instance VI_1 {state BACKUP interface eth0virtual_router_id 100priority 80advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.100/24 dev eth0 label eth0:1}unicast_src_ip 172.25.254.20unicast_peer {172.25.254.10}
}
vrrp_instance VI_2 {state MASTERinterface eth0virtual_router_id 200priority 100advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.200/24 dev eth0 label eth0:2}unicast_src_ip 172.25.254.20unicast_peer {172.25.254.10}
}

2.6实现IPVS的高可用性

# 在realserver1/2
[root@realserver2 ~]# ip a a 172.25.254.100/32 dev lo
[root@realserver2 ~]# vim /etc/sysctl.d/arp.conf
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2
net.ipv4.conf.lo.arp_ignore=1
net.ipv4.conf.lo.arp_announce=2
​
[root@realserver2 ~]# sysctl --system

# ka1/2
[root@ka1 ~]# yum install ipvsadm -y
[root@ka1 ~]# vim /etc/keepalived/keepalived.conf 
virtual_server 172.25.254.100 80 {delay_loop 6lb_algo wrrlb_kind DR#persistence_timeout 50protocol TCP
​real_server 172.25.254.110 80 {weight 1HTTP_GET {url {path /status_code 200}connect_timeout 3nb_get_retry 2delay_before_retry 2}}real_server 172.25.254.120 80 {weight 1HTTP_GET {url {path /status_code 200}connect_timeout 3nb_get_retry 2delay_before_retry 2}}
}

[root@ka1 ~]# systemctl restart keepalived.service  
[root@ka1 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port           Forward Weight ActiveConn InActConn
​
​
[root@ka2 ~]# ipvsadm -A -t 172.255.254.100:80 -s wrr
[root@ka2 ~]# ipvsadm -Ln 
[root@ka1 ~]# ipvsadm -Ln 

# 测试
[root@ka2 ~]# for i in {1..6}; do curl 172.25.254.100; done
realserver2 - 172.25.254.120
realserver1 - 172.25.254.110
realserver2 - 172.25.254.120
realserver1 - 172.25.254.110
realserver2 - 172.25.254.120

2.7实现其它应用的高可用性VRRP Script

keepalived利用 VRRP Script 技术，可以调用外部的辅助脚本进行资源监控，并根据监控的结果实现优先动态调整，从而实现其它应用的高可用性功能

分两步实现：

1、定义脚本
vrrp_script：自定义资源监控脚本，vrrp实例根据脚本返回值，公共定义，可被多个实例调用，定义在vrrp实例之外的独立配置块，一般放在global_defs设置块之后。

通常此脚本用于监控指定应用的状态。一旦发现应用的状态异常，则触发对MASTER节点的权重减至低于SLAVE节点，从而实现 VIP 切换到 SLAVE 节点

vrrp_script <SCRIPT_NAME> {
script <STRING>|<QUOTED-STRING> #此脚本返回值为非0时，会触发下面OPTIONS执行
OPTIONS
}

2、调用脚本
track_script：调用vrrp_script定义的脚本去监控资源，定义在VRRP实例之内，调用事先定义的vrrp_script

track_script {
SCRIPT_NAME_1
SCRIPT_NAME_2
}

2.7.1实战:利用脚本实现主从角色切换

# 在ka1上
[root@ka1 ~]# vim /etc/keepalived/test.sh
#!/bin/bash
[ ! -f /mnt/lee ]
[root@ka1 ~]# sh /etc/keepalived/test.sh 
[root@ka1 ~]# echo $?
0
[root@ka1 ~]# chmod +x /etc/keepalived/test.sh 
[root@ka1 ~]# vim /etc/keepalived/keepalived.conf 
vrrp_script check_haproxy {script "/etc/keepalived/test.sh"interval 1weight -30fall 2rise 2timeout 2
}
vrrp_instance VI_1 {state MASTERinterface eth0virtual_router_id 100priority 100advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.100/24 dev eth0 label eth0:1}unicast_src_ip 172.25.254.10unicast_peer {172.25.254.20}track_script {check_haproxy}
}
[root@ka1 ~]# systemctl restart keepalived.service 

​
# 测试
[root@ka1 ~]# touch /mnt/lee
[root@ka1 ~]# ifconfig
[root@ka2 ~]# ifconfig
eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500inet 172.25.254.100  netmask 255.255.255.0  broadcast 0.0.0.0ether 00:0c:29:44:a9:9f  txqueuelen 1000  (Ethernet)
​
[root@ka1 ~]# rm -rf /mnt/lee
[root@ka1 ~]# systemctl restart keepalived.service 
[root@ka1 ~]# ifconfig
eth0:1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500inet 172.25.254.100  netmask 255.255.255.0  broadcast 0.0.0.0ether 00:0c:29:42:39:a7  txqueuelen 1000  (Ethernet)

2.7.2实战：实现HAProxy高可用

#ka1、ka2
[root@ka1 ~]# yum install haproxy -y
[root@ka1 ~]# vim /etc/sysctl.conf
net.ipv4.ip_nonlocal_bind = 1
[root@ka1 ~]# sysctl -p
[root@ka1 ~]# vim /etc/haproxy/haproxy.cfg
listen webclusterbind 172.25.254.100:80mode httpbalance roundrobinserver web1 172.25.254.110:80 check inter 3 fall 2 rise 5server web2 172.25.254.120:80 check inter 3 fall 2 rise 5
[root@ka1 ~]# sysetemctl enable ---now haproxy
[root@ka1 ~]# netstat -antlupe | grep haproxy
[root@ka1 ~]# curl 172.25.254.100 (不能，和lvs冲突)
[root@ka1 ~]# vim /etc/keepalived/keepalived.conf 
注释掉
#virtual_server 172.25.254.100 80 {
[root@ka1 ~]# systemctl restart keepalived.service  
​
#realwebser1、2
[root@realserver2 ~]# systemctl restart network (删掉lo)
[root@realserver2 ~]# vim /etc/sysctl.d/arp.conf
net.ipv4.conf.all.arp_ignore=0
net.ipv4.conf.all.arp_announce=0
net.ipv4.conf.lo.arp_ignore=0
net.ipv4.conf.lo.arp_announce=0
[root@realserver2 ~]# sysctl --system
​
[root@ka1 ~]# curl 172.25.254.100
[root@ka1 ~]# curl 172.25.254.110
[root@ka1 ~]# curl 172.25.254.120

在ka1关掉haproxy，此时外面访问不了172.25.254.10

# 在ka1
[root@ka1 ~]# vim /etc/keepalived/test.sh
#!/bin/bash
killall -0 haproxy
[root@ka1 ~]# sh /etc/keepalived/test.sh 
[root@ka1 ~]# echo $?
[root@ka1 ~]# vim /etc/keepalived/keepalived.conf
vrrp_script check_haproxy {script "/etc/keepalived/test.sh"interval 1weight -30fall 2rise 2timeout 2
}
vrrp_instance VI_1 {state MASTERinterface eth0virtual_router_id 100priority 100advert_int 1authentication {auth_type PASSauth_pass 1111}virtual_ipaddress {172.25.254.100/24 dev eth0 label eth0:1}unicast_src_ip 172.25.254.10unicast_peer {172.25.254.20}track_script {check_haproxy}
}
[root@ka1 ~]# systemctl restart keepalived.service 
​

# 测试 关掉ka1的haproxy，也能一直访问
[root@ka1 ~]# systemctl stop haproxy.service 
[root@ka1 ~]# systemctl start haproxy.service 
[root@ka2 ~]# while true; do curl 172.25.254.100; done