centos下nvme over rdma 环境配置

nvme over rdma 环境配置

本文主要介绍NVMe over RDMA的安装和配置。关于什么是NVMe over Fabrics,什么是NVMe over RDMA，本文就不做介绍了，网上资料一大堆。
可以看看什么是NVMe over Fabrics？
RDMA（全称：Remote Direct Memory Access）是一种远程直接内存访问技术，通过在硬件中实现传输层协议，将内存/消息原语接口暴露至用户空间，通过绕过CPU和内核网络协议栈来实现高吞吐和低延迟的网络。RoCE（RDMA over Converged Ethernet）是一种允许通过以太网使用远程直接内存访问（RDMA）的网络协议。

前面一文已经提到过centos 7.8 x86_64上面部署rdma环境，下面主要测试一下基于内核态的 nvme of rdma

#show_gids //看看网卡支持的roce版本
DEV     PORT    INDEX   GID                                     IPv4            VER     DEV
---     ----    -----   ---                                     ------------    ---     ---
mlx5_0  1       0       fe80:0000:0000:0000:1270:fdff:fe27:f982                 v1      eth2
mlx5_0  1       1       fe80:0000:0000:0000:1270:fdff:fe27:f982                 v2      eth2
mlx5_0  1       2       0000:0000:0000:0000:0000:ffff:0ac0:3398 10.192.51.152   v1      eth2
mlx5_0  1       3       0000:0000:0000:0000:0000:ffff:0ac0:3398 10.192.51.152   v2      eth2
mlx5_1  1       0       fe80:0000:0000:0000:1270:fdff:fe27:f983                 v1      eth3
mlx5_1  1       1       fe80:0000:0000:0000:1270:fdff:fe27:f983                 v2      eth3
n_gids_found=6

NVMe target配置

#modprobe nvmet
#modprobe nvmet-rdma
#modprobe nvme-rdma

参考前面 nvme of tcp 的步骤

# ls /sys/kernel/config/nvmet/
hosts  ports  subsystems
#cd /sys/kernel/config/nvmet/subsystems
#mkdir nvme-test-target
#tree
.
└── nvme-test-target├── allowed_hosts├── attr_allow_any_host├── attr_serial├── attr_version└── namespaces3 directories, 3 files#cd nvme-test-target/
#允许所有主机访问
#echo 1 > attr_allow_any_host #nvme list
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev  
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     BTAC210600LH3P8AGN   INTEL SSDPF2KX038TZ                      1           3.84  TB /   3.84  TB    512   B +  0 B   JCV10200
/dev/nvme1n1     BTAC210006RS3P8AGN   INTEL SSDPF2KX038TZ                      1           3.84  TB /   3.84  TB    512   B +  0 B   JCV10200
/dev/nvme2n1     BTAC209007EB3P8AGN   INTEL SSDPF2KX038TZ                      1           3.84  TB /   3.84  TB    512   B +  0 B   JCV10200
/dev/nvme3n1     BTAC210007A73P8AGN   INTEL SSDPF2KX038TZ                      1           3.84  TB /   3.84  TB    512   B +  0 B   JCV10200#可以看出/dev/nvme0n1 是namespace 1
#mkdir namespaces/1
#ll namespaces/1
total 0
-rw-r--r-- 1 root root 4096 Sep  3 20:17 device_nguid
-rw-r--r-- 1 root root 4096 Sep  3 20:17 device_path
-rw-r--r-- 1 root root 4096 Sep  3 20:17 device_uuid
-rw-r--r-- 1 root root 4096 Sep  3 20:17 enable

指定namespace 1的device并启用
#echo '/dev/nvme0n1' > namespaces/1/device_path
#echo 1 > namespaces/1/enable

创建端口号配置IP

#mkdir /sys/kernel/config/nvmet/ports/1
#cd /sys/kernel/config/nvmet/ports/1
#ll
total 0
-rw-r--r-- 1 root root 4096 Sep  3 20:21 addr_adrfam
-rw-r--r-- 1 root root 4096 Sep  3 20:21 addr_traddr
-rw-r--r-- 1 root root 4096 Sep  3 20:21 addr_treq
-rw-r--r-- 1 root root 4096 Sep  3 20:21 addr_trsvcid
-rw-r--r-- 1 root root 4096 Sep  3 20:21 addr_trtype
-rw-r--r-- 1 root root 4096 Sep  3 20:21 param_inline_data_size
drwxr-xr-x 2 root root    0 Sep  3 20:21 referrals
drwxr-xr-x 2 root root    0 Sep  3 20:21 subsystems
#echo '10.192.51.152' > addr_traddr #低版本内核 4.19* 下面报过错误，升级5.*后正常
#echo rdma > addr_trtype
#echo 4420 > addr_trsvcid 
#echo ipv4 > addr_adrfam

端口子接口绑定

#ln -s /sys/kernel/config/nvmet/subsystems/nvme-test-target/ /sys/kernel/config/nvmet/ports/1/subsystems/nvme-test-target#dmesg -T| grep "enabling port"
[Fri Sep 13 18:00:05 2024] nvmet_rdma: enabling port 1 (10.192.51.152:4420)

NVMe initiator配置

#yum -y install nvme-cli
#modprobe nvmet
#modprobe nvmet-rdma
#modprobe nvme-rdma
# nvme discover -t rdma -q nvme-test-target -a 10.192.51.152 -s 4420
=====Discovery Log Entry 0======
trtype:  rdma
adrfam:  ipv4
subtype: unrecognized
treq:    not specified, sq flow control disable supported
portid:  1
trsvcid: 4420
subnqn:  nqn.2014-08.org.nvmexpress.discovery
traddr:  10.192.51.152
rdma_prtype: not specified
rdma_qptype: connected
rdma_cms:    rdma-cm
rdma_pkey: 0x0000
=====Discovery Log Entry 1======
trtype:  rdma
adrfam:  ipv4
subtype: nvme subsystem
treq:    not specified, sq flow control disable supported
portid:  1
trsvcid: 4420
subnqn:  nvme-test-target
traddr:  10.192.51.152
rdma_prtype: not specified
rdma_qptype: connected
rdma_cms:    rdma-cm
rdma_pkey: 0x0000
# nvme connect -t rdma  -q nvme-test-target -n nvme-test-target  -a 10.192.51.152 -s 4420
# nvme list #多出来了nvme4n1磁盘
Node             SN                   Model                                    Namespace Usage                      Format           FW Rev  
---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- --------
/dev/nvme0n1     PHAX333201XZ3P8CGN   INTEL SSDPF2KX038T1                      1           3.84  TB /   3.84  TB    512   B +  0 B   9CV10200
/dev/nvme1n1     BTAC209500853P8AGN   INTEL SSDPF2KX038TZ                      1           3.84  TB /   3.84  TB    512   B +  0 B   JCV10200
/dev/nvme2n1     BTAC2095012Z3P8AGN   INTEL SSDPF2KX038TZ                      1           3.84  TB /   3.84  TB    512   B +  0 B   JCV10200
/dev/nvme3n1     BTAC210601LW3P8AGN   INTEL SSDPF2KX038TZ                      1           3.84  TB /   3.84  TB    512   B +  0 B   JCV10200
/dev/nvme4n1     2fb38638ea25c2ad3ee4 Linux                                    1           3.84  TB /   3.84  TB    512   B +  0 B   3.10.0-1