0 Replies Latest reply on Jan 18, 2018 5:28 PM by mpuzio

    How to use NFS over RDMA with MLNX_OFED [solution]

    mpuzio

      Deep buried in MLNX_OFED 4 release notes is a laconic remark that support for NFS over RDMA has been removed. No rationale is provided, and seemingly no one knows why this useful feature was omitted. Fortunately, the release notes turn out to be inaccurate on this point, and the NFS/RDMA support is in fact included in MLNX_OFED, but is not installed by default.

       

      The package that must be installed in addition to the standard MLNX_OFED install is mlnx-nfsrdma-dkms. The procedure below is for Ubuntu 16.04 with kernel 4.13. As of this writing this is the HWE kernel; Ubuntu ISOs are currently distributed with kernel 4.10, which is updated to 4.13 by full upgrade. I tested kernel 4.4 and found that it is not compatible with mlnx-nfsrdma-dkms, so if you have an install with kernel 4.4, you need to replace it with the HWE kernel.

      I assume that a similar solution would work with other versions of Ubuntu and with other Linux distros, but I did not test that. If someone is interested and would like to try, please don't hesitate to post your results.

       

      I tested MLNX_OFED 4.2-1.2.0.0 (newest) and 4.2-1.0.0.0 (previous), with the same success, by installing it from the tarball. This needs to be done on all machines involved with NFS over RDMA, i.e. NFS server(s) and clients.

      Tarball used here is MLNX_OFED_LINUX-4.2-1.2.0.0-ubuntu16.04-x86_64.tgz

       

      First download MLNX_OFED tarball and install it the standard way (here tarball is unpacked to the home directory):

      tar xzf ~/MLNX_OFED_LINUX-4.2-1.2.0.0-ubuntu16.04-x86_64.tgz
      cd ~/MLNX_OFED_LINUX-4.2-1.2.0.0-ubuntu16.04-x86_64/
      ./mlnxofedinstall

       

      The missing package is in source directory of the above tarball. Unpack, build and install it:

      tar xzf ~/MLNX_OFED_LINUX-4.2-1.2.0.0-ubuntu16.04-x86_64/src/MLNX_OFED_SRC-4.2-1.2.0.0.tgz
      cd ~/MLNX_OFED_SRC-4.2-1.2.0.0/
      ./install.pl --kernel-only --build-only
      cd ~/MLNX_OFED_SRC-4.2-1.2.0.0/DEBS/ubuntu16.04/x86_64/
      dpkg -i mlnx-nfsrdma-dkms_3.4-OFED.4.2.1.2.0.1.gf8de107_all.deb

       

      Next, reboot the machine.

       

      To test, export a share on the server, load rpcrdma module and configure rdma port.

      Example (customize to your needs):

      exportfs -o rw,fsid=1,async,no_subtree_check 10.0.0.0/24:/mnt/share/
      modprobe rpcrdma
      echo rdma 20049 > /proc/fs/nfsd/portlist

       

      On the client, load rpcrdma module and mount the share. Example:

      modprobe rpcrdma
      mount -o rdma,port=20049 10.2.0.1:/mnt/share /mnt/mountpoint

       

      Enjoy!