3 Replies Latest reply on Sep 19, 2018 5:43 AM by thull

    When will an ACK generated in RDMA write?

    thull

      Hi,

      Recently, I am doing some RDMA write latency test with ConnectX-4 Lx 25G NIC.

      And I have two questions about the testing:

      1. What is the version of RDMA? Is the default RoCEv2?
      2. By default, the RC(reliable) mode is chosen. As required, an ACK is needed from the remote to local, then the local will add an entry to the CQ and the software then know the data has arrived at the remote. I searched with google but didn't find when a MLNX NIC will generate the ACK. I mean, is it generated when the data received by the HCA or after DMA to the host memory? And I assume the ACK is automatically generated by the NIC adapter without any SW  involvement. Is my understanding right?

       

      Many thanks

        • Re: When will an ACK generated in RDMA write?
          thull

          Could any expert give a hand?

            • Re: When will an ACK generated in RDMA write?
              alkx

              Hi,

              In general, the question is not related to Mellanox, but more to Infiniband SPEC and linux-rdma mailing list could be a better choice. And seems that you already discussing the subject using another forum, which is great by the way - https://www.rdmamojo.com/2013/02/15/ibv_poll_cq/

                • Re: When will an ACK generated in RDMA write?
                  thull

                  Hi Alkx,

                  Many thanks.

                  I will read the specification later when time is available. Yes, I did ask the question on that blog. When I click to commit my question, nothing update there. So I assumed that I failed to commit due to the network or the server's problem. To my understanding, the NIC will generate the ACK once it received the whole frame/segment without any error to accelerate the processing. Indeed, almost all the chips will do so in the embedded platform, maybe the NIC on the server will have the same consideration.

                   

                  I am just confused the cycles gap of poll cq between the RC and UC mode. And by the way, where can I get the latency and BW report of MLNX NICs? Is it public? From the blog, it is said that usually less than 1 microsecond with small packets. But my testing result is more than 1 us. And no matter whether the numactl is used to specify the CPU core and memory allocation.