HowTo Configure Persistent NVMe over Fabrics Initiator

Version 6
Created by maxg on Apr 1, 2018 9:03 AM. Last modified by maxg on Apr 11, 2018 3:03 AM.

    This post provides an example for how to bring up NVMe over Fabrics persistent association between initiator (host) and target, using RDMA transport layer.

    This solution uses systemd services to ensure NVMe-oF initiator remains persistent in case fatal errors occur in the low level device.

     

    >>Learn for free about Mellanox solutions and technologies in the Mellanox Online Academy

     

    Note: This post assumes that the user can configure initial NVMEoF association between target and host, and that the RDMA layer is enabled. Otherwise, please refer to HowTo Configure NVMe over Fabrics.

     

    References

     

    Setup

    • Two servers, one configured as NVMe-oF target, and the other used as host (initiator).

     

    Configuration Video By Mellanox Academy

     

     

    Prerequisites

    1. Configured NVMe-oF target that exposes subsystems over RDMA transport using traddr 1.1.1.1, and trsvcid 4420.

    2. Configured network interfaces on the initiator side that can reach the target portal (e.g. 1.1.1.1:4420). These interfaces should be able to recover fatal errors (e.g. can be configured using BOOTPROTO=static in the ifcfg configuration file).

    3. In order to be persistent also after reboot, one should make sure to load nvme-rdma module in the initiator side at boot time (e.g. by adding nvme-rdma to /etc/modules for debian or by adding nvme-rdma to /etc/modules-load.d/nvme-rdma.conf in RHEL).

    Note: Also need to make sure that the needed network interfaces to reach the target portal (e.g. 1.1.1.1:4420) are configured at boot time as well.

     

    NVMe-oF Persistent Initiator Configuration

     

    Prerequisites

     

    1. Create nvme_fabrics_persistent.service script file under /etc/systemd/system directory:

    [Unit]

    Description=NVMf auto discovery service

    Requires=network.target

    After=systemd-modules-load.service network.target

     

    [Service]

    Type=oneshot

    ExecStart=/usr/sbin/nvme connect-all

    StandardOutput=journal

     

    [Install]

    WantedBy=multi-user.target timers.target

    Note: make sure nvme-cli installation created has nvme executable under /usr/sbin/. Otherwise, locate it using "which nvme" command, and copy the result to ExecStart section in nvme_fabrics_persistent.service script.

     

    2. Create  nvme_fabrics_persistent.timer script file under /etc/systemd/system directory:

    [Unit]

    Description=NVMf auto discovery timer

     

    [Timer]

    OnUnitActiveSec=60

    Unit=nvme_fabrics_persistent.service

     

    [Install]

    WantedBy=multi-user.target timers.target

    Note: In this example, we will trigger the timer every 60 seconds (this can be changed by setting a different value for OnUnitActiveSec attribute).

     

    3. Enable the new systemd service (this will ensure the service is started at boot):

    # systemctl enable nvme_fabrics_persistent.service

     

    4.  Enable new systemd timer (this will ensure the timer is started at boot):

    # systemctl enable nvme_fabrics_persistent.timer

     

    5. Create a configuration file (must be under /etc/nvme/discovery.conf) with the target configuration:

    --transport rdma --traddr 1.1.1.1 --trsvcid 4420

    Note: In this example, we will connect to all the allowed subsystems that are exposed using 1.1.1.1:4420 portal. In order to connect to other subsystems under different portals, you should duplicate the above line and update the relevant attributes. Note that other options (such as --host-traddr/--hostnqn/etc...) can be configured and added to the configuration file.

     

    At this point, NVMe-oF initiator will be durable to HCA fatal errors for the configured portals written in /etc/nvme/discovery.conf file.

    Note: In case you would like to remove a portal from persistent group, the suitable configuration line should be removed from /etc/nvme/discovery.conf file.