2 Replies Latest reply on May 9, 2017 8:10 AM by asantabarbara@siena.edu

    NFS over RDMA on Centos 7.3



      i am trying to install NFS over RDMA on a centos 7.3 with Mellanox.

      I installed MLNX_OFED_LINUX-4.0- and configured a nfs server.

      Kernel: 3.10.0-514.16.1.el7.x86_64

      If i connect to the server via ip, everything is fine. If i try to add rdma support in nfs, i retrieve following errors while starting:

      rpc.nfsd[3423]: rpc.nfsd: Unable to request RDMA services: Protocol not supported


      My lsmod shows:

      lsmod | grep rdma

      rpcrdma                12430  0

      svcrdma                12430  0

      rdma_ucm               22741  0

      rdma_cm                54793  1 rdma_ucm

      iw_cm                  42241  1 rdma_cm

      ib_cm                  51558  3 rdma_cm,ib_ucm,ib_ipoib

      ib_uverbs              75283  2 ib_ucm,rdma_ucm

      ib_core               240474  10 rdma_cm,ib_cm,iw_cm,mlx4_ib,mlx5_ib,ib_ucm,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib

      mlx_compat             16849  16 rdma_cm,ib_cm,iw_cm,rpcrdma,svcrdma,mlx4_en,mlx4_ib,mlx5_ib,ib_ucm,ib_core,ib_umad,ib_uverbs,mlx4_core,mlx5_core,rdma_ucm,ib_ipoib


      Also i tried to add port 20049 manualy to /proc/fs/nfsd/portlist

      echo "rdma 20049" > /proc/fs/nfsd/portlist

      -bash: echo: write error: Protocol not supported


      RDMA tests looks good:

      front01:/etc/infiniband # rdma_server

      rdma_server: start

      rdma_server: end 0


      Speedtests also give results up to 50Gbit.

      Any ideas how i can setup NFS over RDMA or what i am missing?