1. NVMe-oF Target入门指南

SPDK NVMe over Fabrics Target是一个用户空间应用程序，通过以太网，Infiniband或光纤通道等结构呈现块设备,SPDK目前支持RDMA和TCP传输。

本文将在已经编译好SPDK的基础上演示如何使用SPDK搭建NVMe over TCP，前提是您已经将一块NVMe硬盘挂载到linux系统中。

如果您想获取有关SPDK更详细的信息，请参考官方文档。

演示过程的Host主机的ip地址是192.168.0.21，Target主机的ip地址是192.168.0.22

1.1 Target端配置

假设您已经成功编译SPDK

初始化SPDK

# 进入spdk目录
$ cd spdk

# 运行脚本转让设备控制权给SPDK
$ sudo scripts/setup.sh
0000:03:00.0 (15ad 07f0): nvme -> uio_pci_generic

# 查看设备状态
$ sudo scripts/setup.sh status
Hugepages
node     hugesize     free /  total
node0   1048576kB        0 /      0
node0      2048kB     1024 /   1024

Type     BDF             Vendor Device NUMA    Driver           Device     Block devices
NVMe     0000:03:00.0    15ad   07f0   0       uio_pci_generic  -          -

启动 nvmf-tgf

1	$ sudo build/bin/nvmf_tgt

创建TCP传输

1	$ sudo scripts/rpc.py nvmf_create_transport -t TCP -u 16384 -m 8 -c 8192

创建 NVMe 块设备

1	$ sudo scripts/rpc.py bdev_nvme_attach_controller -b NVMe1 -t PCIe -a 0000:03:00.0

将NVMe块设备分配给子系统

# 创建子系统
$ sudo scripts/rpc.py nvmf_create_subsystem nqn.2016-06.io.spdk:cnode1 -a -s SPDK00000000000001 -d SPDK_Controller1

# 将NVMe块设备分配给刚刚创建的子系统
$ sudo scripts/rpc.py nvmf_subsystem_add_ns nqn.2016-06.io.spdk:cnode1 NVMe1n1

# 为子系统添加TCP监听器
$ sudo scripts/rpc.py nvmf_subsystem_add_listener nqn.2016-06.io.spdk:cnode1 -t TCP -a 192.168.0.22 -s 4420

1.2 Host端配置

安装 nvme-cli

1	$ sudo apt install nvme-cli

加载驱动程序

1	$ sudo modprobe nvme-tcp

发现 Target

$ sudo nvme discover -t tcp -a 192.168.0.22 -s 4420
Discovery Log Number of Records 1, Generation counter 5
=====Discovery Log Entry 0======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not required
portid:  0
trsvcid: 4420
subnqn:  nqn.2016-06.io.spdk:cnode1
traddr:  192.168.0.22
sectype: none

连接Target

$ sudo nvme connect -t tcp -n "nqn.2016-06.io.spdk:cnode1" -a 192.168.0.22 -s 4420

# 查看设备方法1
$ sudo fdisk -l
...
Disk /dev/nvme0n1: 5 GiB, 5368709120 bytes, 10485760 sectors
Disk model: SPDK_Controller1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x441af706

Device         Boot Start      End  Sectors Size Id Type
/dev/nvme0n1p1       2048 10485759 10483712   5G 83 Linux
...

# 查看设备方法2
$ lsblk
NAME                      MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
fd0                         2:0    1    4K  0 disk 
loop0                       7:0    0 55.5M  1 loop /snap/core18/2074
loop2                       7:2    0 67.6M  1 loop /snap/lxd/20326
loop3                       7:3    0 32.3M  1 loop /snap/snapd/11107
loop4                       7:4    0 32.3M  1 loop /snap/snapd/12398
loop5                       7:5    0 55.5M  1 loop /snap/core18/1988
loop6                       7:6    0 70.3M  1 loop /snap/lxd/21029
sda                         8:0    0   20G  0 disk 
├─sda1                      8:1    0    1M  0 part 
├─sda2                      8:2    0    1G  0 part /boot
└─sda3                      8:3    0   19G  0 part 
  └─ubuntu--vg-ubuntu--lv 253:0    0   19G  0 lvm  /
sr0                        11:0    1 1024M  0 rom  
sr1                        11:1    1  1.1G  0 rom  
nvme0n1                   259:1    0   20G  0 disk

此时可将块设备nvme0n1p1进行挂载访问

1 2	# 将/dev/nvme0n1挂载到/root/data/下 $ mount -t ext4 -o rw /dev/nvme0n1 /root/data/

取消连接Target

$ nvme disconnect -n "nqn.2016-06.io.spdk:cnode1"

# 或者
$ nvme disconnect-all

Reference
[1]: https://spdk.io/doc/nvmf.html
[2]: https://www.cnblogs.com/JamesLi/p/11399054.html

← 使用SPDK部署iSCSI A Community Cache with Complete Information 论文阅读笔记 →

使用SPDK部署NVMe over TCP

1. NVMe-oF Target入门指南

1.1 Target端配置

1.2 Host端配置