3 Replies Latest reply on Aug 24, 2017 12:22 AM by aegq

    NFSoRDMA: svcrdma: Error -12 posting RDMA_READ

    aegq

      Hi

          I configured the NFSoRDMA according to this document.  HowTo Configure NFS over RDMA (RoCE)

          This is the same problem, but it does not help me.   ConnectX-4 LX: NFSoRDMA failures in MOFED-3.3 

       

      Here's my question:

           After the configuration, start fio testing,The read test will succeed and the write test will fail !

       

      INFO:  NFS SERVER

      image.png

       

      INFO: NFS CLIENT:

      image.png

       

      Any suggestions on it ?

      Regards

         guoqingwang

        • Re: NFSoRDMA: svcrdma: Error -12 posting RDMA_READ
          aegq

          Recently,

          I tried to upgrade the kernel to 3.16.46 ( MLNX_OFED_LINUX-3.4-1.0.0.0 ) , but it did not work.

          I tried to upgrade the driver to MLNX_OFED_LINUX-4.1-1.0.2.0 (kernel: 3.16.46), but The kernel module (xprtrdma) does not seem to match the driver version.

           

           

          alkx  Can you help me and give me some tips ?  Thank you 

            • Re: NFSoRDMA: svcrdma: Error -12 posting RDMA_READ
              martijn@mellanox.com

              Hi guoqingwang,

               

              Thank you for reaching out to the Mellanox Support Community and for your patience in this matter.

               

              Unfortunately, as from Mellanox OFED version 3.4-x and above NFSoRDMA is not supported anymore. This was never mentioned in the Release Notes for version 3.4-1.0.0.0 but it is mentioned in later released version.

              If you want to use NFSoRDMA, you need to downgrade the driver to Mellanox OFED version 3.3-1.0.4.0 in which we do still support NFSoRDMA and is available for Ubuntu 14.04 or switch to the OS-vendor supplied driver (INBOX Driver) for the Mellanox NIC.

               

              Hopefully this will clarify and resolves your issue.

               

              Thanks.

               

              Cheers,

              ~Martijn

                • Re: NFSoRDMA: svcrdma: Error -12 posting RDMA_READ
                  aegq

                  Hi  Martijn :

                   

                       thank you very much for your reply

                   

                       After downgraded the driver to Mellanox OFED version 3.3-1.0.4.0 from your advice, I have recompiled the mlnx-ofed-kernel module with nfsrdma support .

                       But still get same error  "svcrdma: Error -12 posting RDMA_READ" while executing “fio --rw=randwrite“.

                       I wonder if there is a bug or some wrong configuration .

                   

                       Hope to get any of your suggestions .

                   

                  Thanks.

                   

                  Regards

                     guoqingwang

                   

                   

                  PS:

                  HowTo Compile MLNX_OFED Drivers (mlnx-ofa_kernel example)

                  options taken from /etc/infiniband/info + --with-nfsrdma-mod

                  mlnx-ofed-kernel-3.3# ./configure --with-core-mod --with-user_mad-mod --with-user_access-mod --with-addr_trans-mod --with-mlx4-mod --with-mlx4_en-mod --with-mlx5-mod --with-ipoib-mod --with-sdp-mod --with-e_ipoib-mod --with-nfsrdma-mod

                  mlnx-ofed-kernel-3.3# make

                   

                  copy xprtrdma.ko to nfs client and svcrdma.ko to nfs server

                  mlnx-ofed-kernel-3.3# cp net/sunrpc/xprtrdma/xprtrdma.ko /lib/modules/3.16.46-031646-generic/updates/dkms/

                  mlnx-ofed-kernel-3.3# depmod -a && modprobe xprtrdma