lvs dr+keepalived

基于keepalived(主从+双主) + LVS(DR模型) + DNS实现http高可用集群

keepalived高可用主机IP：172.21.5.22和172.21.5.21
http服务高可用主机IP：172.21.5.16和172.21.5.18
VIP采用172.16.32.5

各虚拟机及主机名和IP对应关系如下所示：

虚拟机	主机名	IP地址
HA1	node1	172.21.5.22
HA2	node2	172.21.5.21
HA3	node3	172.21.5.16
HA4	node4	172.21.5.18

接下来我们先去配置好http高可用服务两台主机，然后再回来配置keepalived。既然是LVS的DR模型，那我们先来配置另外两台主机，来实现http高可用集群。
首先，打开另外两外两台提供http服务的虚拟机，这里是HA3和HA4。为了方便你也可以改为RS1和RS2。

在DR模型中，只添加个http服务还不行，还有许多需要修改，貌似不是很简单，好吧，为了节约时间，这里我们通过一个脚本实现修改各个数据。

#!/bin/bash
#
# Script to start LVS DR real server.
# chkconfig: - 90 10
# description: LVS DR real server
#. /etc/rc.d/init.d/functionsVIP=172.16.32.5 # 定义VIP
host=$(/bin/hostname)case "$1" in
start)# Start LVS-DR real server on this machine./sbin/ifconfig lo down/sbin/ifconfig lo upecho 1 > /proc/sys/net/ipv4/conf/lo/arp_ignoreecho 2 > /proc/sys/net/ipv4/conf/lo/arp_announceecho 1 > /proc/sys/net/ipv4/conf/all/arp_ignoreecho 2 > /proc/sys/net/ipv4/conf/all/arp_announce/sbin/ifconfig lo:0 $VIP broadcast $VIP netmask 255.255.255.255 up/sbin/route add -host $VIP dev lo:0;;
stop)# Stop LVS-DR real server loopback device(s)./sbin/ifconfig lo:0 downecho 0 > /proc/sys/net/ipv4/conf/lo/arp_ignoreecho 0 > /proc/sys/net/ipv4/conf/lo/arp_announceecho 0 > /proc/sys/net/ipv4/conf/all/arp_ignoreecho 0 > /proc/sys/net/ipv4/conf/all/arp_announce;;
status)# Status of LVS-DR real server.islothere=$(/sbin/ifconfig lo:0 | grep $VIP)isrothere=$(netstat -rn | grep "lo:0" | grep $VIP)if [ ! "$islothere" -o ! "$isrothere" ]; then# Either the route or the lo:0 device not found.echo "LVS-DR real server Stopped."elseecho "LVS-DR real server Running."fi;;
*)# Invalid entry.echo "$0: Usage: $0 {start|status|stop}"exit 1;;
esac

记得该脚本需要在另一个主机上也要执行一下。
两台主机上都执行过上边的脚本后，验证下各参数是否已经修改：下面几个命令均在node3主机上执行，可在node3上使用ssh node4 'COMMAND’来验证下node4上各参数是否已修改

[root@wh ~]# ifconfig    #在node3上执行该命令，查看是否有VIP，使用ssh node4 'ifconfig'命令查看node4上是否也有VIP
eth0 Link encap:Ethernet  HWaddr 00:0C:29:7F:8F:44inet addr:172.21.5.18Bcast:172.16.255.255Mask:255.255.0.0UP BROADCAST RUNNING MULTICAST  MTU:1500Metric:1RX packets:162748errors:0dropped:0overruns:0frame:0TX packets:2368errors:0dropped:0overruns:0carrier:0collisions:0txqueuelen:1000RX bytes:26502458(25.2MiB)  TX bytes:200681(195.9KiB)Interrupt:59Base address:0x2000
lo  Link encap:Local Loopbackinet addr:127.0.0.1Mask:255.0.0.0UP LOOPBACK RUNNING  MTU:16436Metric:1RX packets:10errors:0dropped:0overruns:0frame:0TX packets:10errors:0dropped:0overruns:0carrier:0collisions:0txqueuelen:0RX bytes:666(666.0b)  TX bytes:666(666.0b)lo:0Link encap:Local Loopbackinet addr:172.16.32.5Mask:255.255.255.255UP LOOPBACK RUNNING  MTU:16436Metric:1
[root@wh ~]# route -n   #查看是否有配置的VIP特定路由
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
172.16.32.50.0.0.0255.255.255.255UH    000lo
169.254.0.00.0.0.0255.255.0.0U     000eth0
172.16.0.00.0.0.0255.255.0.0U     000eth0
0.0.0.0172.16.0.10.0.0.0UG    000eth0
[root@wh ~]# cat /proc/sys/net/ipv4/conf/all/arp_ignore
1
[root@wh ~]# cat /proc/sys/net/ipv4/conf/all/arp_announce
2

到此，我们的两台RealServer都以配置完毕，而且其http服务也已正常工作。

现在去编辑我们的keepalived的配置文件，并修改成我们所需要的。

#先在node1主机上进行修改：
[root@wh ~]# vim /etc/keepalived/keepalived.conf
global_defs {}vrrp_script chk_httpd {script "killall -0 httpd"interval 2weight -2fall 2rise 1
}vrrp_script chk_schedown {script "[[ -f /etc/keepalived/down ]] && exit 1 || exit 0"interval 2weight -2
}vrrp_instance VI_1 {state MASTERinterface eth0virtual_router_id 132priority 101advert_int 1authentication {auth_type PASSauth_pass langdu}virtual_ipaddress {172.16.32.5/16 dev eth0 label eth0:0}track_script {chk_httpdchk_schedown}notify_master "/etc/keepalived/notify.sh master"notify_backup "/etc/keepalived/notify.sh backup"notify_fault "/etc/keepalived/notify.sh fault"
}virtual_server 172.16.32.5 80 {delay_loop 6lb_algo rrlb_kind DRnat_mask 255.255.0.0protocol TCPreal_server 172.21.5.16 80 {weight 1HTTP_GET {url {path /status_code 200}connect_timeout 2nb_get_retry 3delay_before_retry 2}}real_server 172.21.5.18 80 {weight 2HTTP_GET {url {path /status_code 200}connect_timeout 2nb_get_retry 3delay_before_retry 3}}
}

下面这个脚本时实现健康检查用的。即上边用到的notify.sh脚本

#!/bin/bash
# Author: onlyyou
# description: 一个通知脚本示例ifalias=${2:-eth0:0}
interface=$(echo $ifalias | awk -F: '{print $1}')
vip=$(ip addr show $interface | grep $ifalias | awk '{print $2}')
contact='root@localhost'
workspace=$(dirname $0)notify() {subject="$vip 地址切换为 $1"body="$vip 地址切换为 $1 $(date '+%F %H:%M:%S')"echo $body | mail -s "$1 过渡" $contact # 实现发送邮件
}case "$1" in
master)notify masterexit 0;;
backup)notify backup/etc/rc.d/init.d/httpd restartexit 0;;
fault)notify faultexit 0;;
*)echo "用法: $(basename $0) {master|backup|fault}"exit 1;;
esac

scp /etc/keepalived/keepalived.conf node2:/etc/keepalived/
# 将配置文件发给另一个keepalived主机，这里发给node2，假如你的主机不是node2，请做相应修改
# 在node2主机上，修改刚传过来的keepalived的配置文件。这里只需修改两项即可：
state BACKUP          #设置该node2主机为backup路由priority 100 #设定node2主机的优先级为100，低于node1
# 在node2主机上，我们只需修改上述两项即可。

修改完成后保存退出

现在我们去安装ipvsadm软件包，然后再启动keepalived服务

ssh node2 'yum -y install ipvsadm'   #通过node1主机在node2上安装ipvsadm软件包
yum -y install ipvsadm   #在node1上安装软件包
安装完毕后启动我们的keepalived服务。
service keepalived start    #启动node1上的keepalived服务
ssh node2 'service keepalived start'   #在node1上启动node2的keepalived服务

查看下ipvs规则

ipvsadm -l -n   #使用该命令，查看是否有ipvs规则，显示如下：
IP Virtual Server version 1.2.1(size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  172.16.32.5:80rr
-> 172.21.5.18:80Route   100
-> 172.21.5.16:80Route   100

ifconfig    #查看下node1上的IP配置情况
eth0      Link encap:Ethernet  HWaddr 00:0C:29:9F:2F:AFinet addr:172.21.5.22Bcast:172.16.255.255Mask:255.255.0.0UP BROADCAST RUNNING MULTICAST  MTU:1500Metric:1RX packets:290653errors:1dropped:0overruns:0frame:0TX packets:13874errors:0dropped:0overruns:0carrier:0collisions:0txqueuelen:1000RX bytes:46393162(44.2MiB)  TX bytes:2014631(1.9MiB)Interrupt:59Base address:0x2000
eth0:0Link encap:Ethernet  HWaddr 00:0C:29:9F:2F:AFinet addr:172.16.32.5Bcast:0.0.0.0Mask:255.255.0.0UP BROADCAST RUNNING MULTICAST  MTU:1500Metric:1Interrupt:59Base address:0x2000
lo        Link encap:Local Loopbackinet addr:127.0.0.1Mask:255.0.0.0UP LOOPBACK RUNNING  MTU:16436Metric:1RX packets:10errors:0dropped:0overruns:0frame:0TX packets:10errors:0dropped:0overruns:0carrier:0collisions:0txqueuelen:0RX bytes:666(666.0b)  TX bytes:666(666.0b)

配置的VIP也已生效，我们的ipvs规则已经实现。现在在我们的物理机上访问下172.16.32.5，看一下，是否可以访问，显示什么信息吧。

到目前为止，貌似我们还没有实现IP地址漂移。我们只需在master路由主机上，在相应目录下创建一个down文件即可实现手动漂移IP地址。

在node1上

cd /etc/keepalived/#进入该目录
touch down    #创建该文件，用来实现手动漂移IP地址

tail /var/log/messages    #停几秒钟后，查看日志
May 1619:15:32node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Entering MASTER STATE
May 1619:15:32node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) setting protocol VIPs.
May 1619:15:32node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for172.16.32.5
May 1619:15:32node1 Keepalived_healthcheckers[2815]: Netlink reflector reports IP 172.16.32.5added
May 1619:15:32node1 Keepalived_vrrp[2816]: Netlink reflector reports IP 172.16.32.5added
May 1619:15:32node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Received higher prio advert    #收到更高优先级的通告信息
May 1619:15:32node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Entering BACKUP STATE     #进入backup状态
May 1619:15:32node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) removing protocol VIPs.     #转移VIP地址
May 1619:15:32node1 Keepalived_healthcheckers[2815]: Netlink reflector reports IP 172.16.32.5removed
May 1619:15:32node1 Keepalived_vrrp[2816]: Netlink reflector reports IP 172.16.32.5removed

ifconfig   #使用该命令，查看下node1主机的VIP是否存在，可看到已转移到其他主机
eth0      Link encap:Ethernet  HWaddr 00:0C:29:9F:2F:AFinet addr:172.21.5.22Bcast:172.16.255.255Mask:255.255.0.0UP BROADCAST RUNNING MULTICAST  MTU:1500Metric:1RX packets:347881errors:1dropped:0overruns:0frame:0TX packets:21333errors:0dropped:0overruns:0carrier:0collisions:0txqueuelen:1000RX bytes:50760780(48.4MiB)  TX bytes:2653767(2.5MiB)Interrupt:59Base address:0x2000
lo        Link encap:Local Loopbackinet addr:127.0.0.1Mask:255.0.0.0UP LOOPBACK RUNNING  MTU:16436Metric:1RX packets:10errors:0dropped:0overruns:0frame:0TX packets:10errors:0dropped:0overruns:0carrier:0collisions:0txqueuelen:0RX bytes:666(666.0b)  TX bytes:666(666.0b)> 在node2主机上查看其日志。
```bash
tail /var/log/messages   #查看node2的日志信息
May 1619:15:32node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) forcing a new MASTER election
May 1619:15:32node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) forcing a new MASTER election
May 1619:15:33node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) Transition to MASTER STATE
May 1619:15:34node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) Entering MASTER STATE    #进入master状态
May 1619:15:34node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) setting protocol VIPs.    #设定VIP
May 1619:15:34node1 Keepalived_healthcheckers[2463]: Netlink reflector reports IP 172.16.32.5added
May 1619:15:34node1 avahi-daemon[3375]: Registering new address record for172.16.32.5on eth0.
May 1619:15:34node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for172.16.32.5
May 1619:15:34node1 Keepalived_vrrp[2464]: Netlink reflector reports IP 172.16.32.5added
May 1619:15:39node1 Keepalived_vrrp[2464]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for172.16.32.5

在在我们的物理机访问下，看能否访问。依然在浏览器地址栏输入172.16.32.5，可以看到，访问正常。
现在，我们删掉/etc/keepalived/down这个文件，看能否实现IP漂移回来。

rm -f /etc/keepalived/down      #删除node1主机上该文件

tail /var/log/messages    #查看日志信息
May 1619:15:32node1 Keepalived_vrrp[2816]: Netlink reflector reports IP 172.16.32.5removed
May 1619:27:54node1 Keepalived_vrrp[2816]: VRRP_Script(chk_schedown) succeeded
May 1619:27:55node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) forcing a new MASTER election
May 1619:27:55node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) forcing a new MASTER election
May 1619:27:56node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Transition to MASTER STATE
May 1619:27:57node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Entering MASTER STATE    #进入master状态
May 1619:27:57node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) setting protocol VIPs.
May 1619:27:57node1 Keepalived_vrrp[2816]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for172.16.32.5
May 1619:27:57node1 Keepalived_healthcheckers[2815]: Netlink reflector reports IP 172.16.32.5added
May 1619:27:57node1 Keepalived_vrrp[2816]: Netlink reflector reports IP 172.16.32.5added

至此，我们已成功实现了keepalived的相关功能。以上演示的仅仅是主从模式下地址漂移。那我们能否实现在双主模式下实现地址漂移呢？答案是肯定的。