HowTo Configure NVMe over Fabrics with Multipath

Version 1

    This post gives a basic configuration of multipath with NVMe over Fabrics devices

     

    >>Learn for free about Mellanox solutions and technologies in the Mellanox Online Academy

     

    Note: This post assumes that the user can configure initial NVMEoF association between target and host, and that the RDMA layer is enabled. Otherwise, please refer to HowTo Configure NVMe over Fabrics.

     

    References

     

    Setup

    • Two servers, one configured as NVMe-oF target, and the other used as host (initiator).

     

    Configuration Video By Mellanox Academy

     

     

    NVMe-oF Initiator Multipath Configuration

     

    Prerequisites

    • multipath is installed.

     

    1. Add the following to /etc/multipath.conf under 'defaults' block:

    uid_attribute         ID_WWN

     

    2. Add the following to /etc/multipath.conf under 'blacklist_exceptions' block (if not created - create one yourself):

    blacklist_exceptions {

      property "(ID_WWN|SCSI_IDENT_.*|ID_SERIAL|DEVTYPE)"

      devnode "nvme*"

    }

     

    3. Add the following to udev rules under /etc/udev/rules.d/<any_udev_rules_file> or /lib/udev/rules.d/60-persistent-storage.rules:

    KERNEL=="nvme*[0-9]n*[0-9]", ENV{DEVTYPE}=="disk", ATTRS{wwid}=="?*", ENV{ID_WWN}="$attr{wwid}"

    Note: To load the new udev rules run:

    # udevadm control --reload-rules && udevadm trigger

    Verification

    • Configure a nvme target with a subsystem that is exposed by at least two ports.
    • After connecting to the subsystem from all its ports make sure multipathd service is active by running:

    # service multipathd status

    • rescan the new nvme devices by running:

    # multipath && multipathd reconfigure

    Now you should see the new device mapper by running:

    # multipath -ll

    mpathfdj (nvme.0000-38346333313765313530643532663832-4c696e7578-00000001) dm-0 NVME,Linux

    size=20G features='0' hwhandler='0' wp=rw

    `-+- policy='service-time 0' prio=1 status=active

      |- #:#:#:# nvme0n1 259:0 active ready running

      `- #:#:#:# nvme1n1 259:1 active ready running

     

    That's it

     

    Troubleshooting

    • In case multipath doesn't work properly you can run multipath -v3, the output will show you its recent actions and errors.
    • In order to use DM multipath with upstream kernel (and NVMeoF), make sure that CONFIG_NVME_MULTIPATH is not set in your kernel configuration.
    • Starting from MLNX_OFED 4.2 and above, path_checker "tur" is no longer supported, so one should replace it with "directio" in the local /etc/multipath.conf file.
    • A basic /etc/multipath.conf file example:

    # cat /etc/multipath.conf

    defaults {

        user_friendly_names     yes

        find_multipaths              no

        max_fds                        65535

        uid_attribute                  ID_WWN

        path_checker                directio

    }

     

    blacklist_exceptions {

      property "(ID_WWN|SCSI_IDENT_.*|ID_SERIAL|DEVTYPE)"

      devnode "nvme*"

    }