DPVS-2:单臂负载均衡测试

上一篇编译安装了DPVS，这一篇开启DPVS的负载均衡测试 ： 单臂 + FULL NAT模式

拓扑-单臂

单臂模式
DPVS 单独物理机
CLINET，和两个RS都是另一个物理机的虚拟机，它们网卡都绑定在一个桥上br0 ， 二层互通。

启动DPVS

dpvs.conf 使用默认的dpvs.conf.single-nic.sample的配置

root@r750-132:~/dpvs/bin#  ./dpvs -- -a 98:00.1 -l 0-9 
current thread affinity is set to FFFFFFFF
EAL: Detected 32 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:98:00.1 (socket 1)
EAL: No legacy callbacks, legacy socket not created
DPVS: dpvs version: 1.9-8, build on 2025.02.20.15:37:58
DPVS: dpvs-conf-file: /etc/dpvs.conf
DPVS: dpvs-pid-file: /var/run/dpvs.pid
DPVS: dpvs-ipc-file: /var/run/dpvs.ipc
CFG_FILE: Opening configuration file '/etc/dpvs.conf'.
CFG_FILE: log_level = WARNING
NETIF: dpdk0:rx_queue_number = 8
NETIF: worker cpu1:dpdk0 rx_queue_id += 0
NETIF: worker cpu1:dpdk0 tx_queue_id += 0
NETIF: worker cpu2:dpdk0 rx_queue_id += 1
NETIF: worker cpu2:dpdk0 tx_queue_id += 1
NETIF: worker cpu3:dpdk0 rx_queue_id += 2
NETIF: worker cpu3:dpdk0 tx_queue_id += 2
NETIF: worker cpu4:dpdk0 rx_queue_id += 3
NETIF: worker cpu4:dpdk0 tx_queue_id += 3
NETIF: worker cpu5:dpdk0 rx_queue_id += 4
NETIF: worker cpu5:dpdk0 tx_queue_id += 4
NETIF: worker cpu6:dpdk0 rx_queue_id += 5
NETIF: worker cpu6:dpdk0 tx_queue_id += 5
NETIF: worker cpu7:dpdk0 rx_queue_id += 6
NETIF: worker cpu7:dpdk0 tx_queue_id += 6
NETIF: worker cpu8:dpdk0 rx_queue_id += 7
NETIF: worker cpu8:dpdk0 tx_queue_id += 7
SAPOOL: sapool_filter_enable = on
IPVS: dp_vs_conn_init: lcore 9: nothing to do.
NETIF: Ethdev port_id=0 invalid tx_offload: 0x1000e, valid value: 0xc96af

配置指令

VIP=192.168.100.100
LIP=192.168.100.200
RS1=192.168.100.4
RS2=192.168.100.5# 为接口配置一个虚拟IP
./dpip addr add 192.168.100.100/24 dev dpdk0# 添加一个虚拟服务，地址为VIP，端口80，负载均衡策略为 轮询(rr= Round Robin)
./ipvsadm -A -t 192.168.100.100:80 -s rr# 为虚拟服务添加一个真实服务器， -b FullNAT模式(源目的地址都转换) 
./ipvsadm -a -t 192.168.100.100:80 -r 192.168.100.4:80 -b
./ipvsadm -a -t 192.168.100.100:80 -r 192.168.100.5:80 -b# 为虚拟服务添加一个本地地址，FNAT之后访问真实服务器的源地址
./ipvsadm --add-laddr -z 192.168.100.200 -t 192.168.100.100:80 -F dpdk0

查看配置

root@r750-132:~/dpvs/bin# ./ipvsadm -L -n
IP Virtual Server version 1.9.8 (size=0)
Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.100.100:80 rr-> 192.168.100.4:80             FullNat 1      0          0         -> 192.168.100.5:80             FullNat 1      0          0   

服务器配置

RS host 192.168.100.4 中，nginx返回值中设为 1

RS host 192.168.100.5中，nginx 返回值中设为 2

root@ubuntu22-1:~# cat /var/www/html/index.nginx-debian.html 
This is Server 1 !root@ubuntu22-2:~# cat /var/www/html/index.nginx-debian.html
This is Server 2 !

服务器ping 负载均衡器，连同正常

root@ubuntu22:~# ping 192.168.100.100
PING 192.168.100.100 (192.168.100.100) 56(84) bytes of data.
64 bytes from 192.168.100.100: icmp_seq=1 ttl=64 time=0.381 ms
64 bytes from 192.168.100.100: icmp_seq=2 ttl=64 time=0.106 ms
^C
--- 192.168.100.100 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1026ms
rtt min/avg/max/mdev = 0.106/0.243/0.381/0.137 ms
root@ubuntu22:~# ping 192.168.100.200
PING 192.168.100.200 (192.168.100.200) 56(84) bytes of data.
64 bytes from 192.168.100.200: icmp_seq=1 ttl=64 time=0.295 ms
64 bytes from 192.168.100.200: icmp_seq=2 ttl=64 time=0.125 ms
^C
--- 192.168.100.200 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1005ms
rtt min/avg/max/mdev = 0.125/0.210/0.295/0.085 ms

CURL测试

测试成功

负载均衡轮询模式看起来生效了。

但是存在有连续的数字，可能与curl测试同时只有一个连接有关，或者每一次被RSS分配到不同的worker有关，具体后续再读源码分析。

root@ubuntu22:~# curl 192.168.100.100:80
This is Server 2 !
root@ubuntu22:~# curl 192.168.100.100:80
This is Server 1 !
root@ubuntu22:~# curl 192.168.100.100:80
This is Server 1 !
root@ubuntu22:~# curl 192.168.100.100:80
This is Server 1 !
root@ubuntu22:~# curl 192.168.100.100:80
This is Server 2 !
root@ubuntu22:~# curl 192.168.100.100:80
This is Server 2 !
root@ubuntu22:~# curl 192.168.100.100:80
This is Server 1 !
root@ubuntu22:~# curl 192.168.100.100:80
This is Server 1 !
root@ubuntu22:~# curl 192.168.100.100:80
This is Server 2 !
root@ubuntu22:~# curl 192.168.100.100:80
This is Server 2 !
root@ubuntu22:~# curl 192.168.100.100:80
This is Server 2 !
root@ubuntu22:~# curl 192.168.100.100:80
This is Server 1 !
root@ubuntu22:~# curl 192.168.100.100:80
This is Server 1 !
root@ubuntu22:~# curl 192.168.100.100:80
This is Server 1 !
root@ubuntu22:~# curl 192.168.100.100:80
This is Server 1 !

查看邻居表

这里会显示每一个worker的邻居表， 当有一个邻居更新时，会同步到所有的worker。

root@r750-132:~/dpvs/bin# ./dpip neigh show dev dpdk0
ip: 192.168.100.3                                    mac: 52:54:00:b3:0d:e5   state: REACHABLE     dev: dpdk0  core: 1  
ip: 192.168.100.4                                    mac: 52:54:00:77:2b:73   state: REACHABLE     dev: dpdk0  core: 1  
ip: 192.168.100.5                                    mac: 52:54:00:8c:53:e9   state: REACHABLE     dev: dpdk0  core: 1  
ip: 192.168.100.3                                    mac: 52:54:00:b3:0d:e5   state: REACHABLE     dev: dpdk0  core: 3  
ip: 192.168.100.4                                    mac: 52:54:00:77:2b:73   state: REACHABLE     dev: dpdk0  core: 3  
ip: 192.168.100.5                                    mac: 52:54:00:8c:53:e9   state: REACHABLE     dev: dpdk0  core: 3  
ip: 192.168.100.3                                    mac: 52:54:00:b3:0d:e5   state: REACHABLE     dev: dpdk0  core: 2  
ip: 192.168.100.4                                    mac: 52:54:00:77:2b:73   state: REACHABLE     dev: dpdk0  core: 2  
ip: 192.168.100.5                                    mac: 52:54:00:8c:53:e9   state: REACHABLE     dev: dpdk0  core: 2  
ip: 192.168.100.3                                    mac: 52:54:00:b3:0d:e5   state: REACHABLE     dev: dpdk0  core: 4  
ip: 192.168.100.4                                    mac: 52:54:00:77:2b:73   state: REACHABLE     dev: dpdk0  core: 4  
ip: 192.168.100.5                                    mac: 52:54:00:8c:53:e9   state: REACHABLE     dev: dpdk0  core: 4  
ip: 192.168.100.3                                    mac: 52:54:00:b3:0d:e5   state: REACHABLE     dev: dpdk0  core: 5  
ip: 192.168.100.4                                    mac: 52:54:00:77:2b:73   state: REACHABLE     dev: dpdk0  core: 5  
ip: 192.168.100.5                                    mac: 52:54:00:8c:53:e9   state: REACHABLE     dev: dpdk0  core: 5  
ip: 192.168.100.3                                    mac: 52:54:00:b3:0d:e5   state: REACHABLE     dev: dpdk0  core: 6  
ip: 192.168.100.4                                    mac: 52:54:00:77:2b:73   state: REACHABLE     dev: dpdk0  core: 6  
ip: 192.168.100.5                                    mac: 52:54:00:8c:53:e9   state: REACHABLE     dev: dpdk0  core: 6  
ip: 192.168.100.3                                    mac: 52:54:00:b3:0d:e5   state: REACHABLE     dev: dpdk0  core: 7  
ip: 192.168.100.4                                    mac: 52:54:00:77:2b:73   state: REACHABLE     dev: dpdk0  core: 7  
ip: 192.168.100.5                                    mac: 52:54:00:8c:53:e9   state: REACHABLE     dev: dpdk0  core: 7  
ip: 192.168.100.3                                    mac: 52:54:00:b3:0d:e5   state: REACHABLE     dev: dpdk0  core: 8  
ip: 192.168.100.4                                    mac: 52:54:00:77:2b:73   state: REACHABLE     dev: dpdk0  core: 8  
ip: 192.168.100.5                                    mac: 52:54:00:8c:53:e9   state: REACHABLE     dev: dpdk0  core: 8  

抓包验证

在RS1上抓包，请求SRC IP已经是DPVS的 LocalIP， DST IP为RS1 IP， 确实是FULL NAT模式。