HowTo Set the Default RoCE Mode When Using RDMA CM

Version 4

    This post describes how to change the default configuration of RoCE mode for RDMA CM applications using the cma_roce_mode tool.

    Note: This is relevant for MLNX_OFED v3.4 and above.

     

    References

     

    Setup Example

    • Two servers equipped with ConnectX-4 adapters connected to a Spectrum switch.
    • Additional server can be connected to Spectrum for monitoring (e.g. install wireshark and capture the traffic).
    • MLNX_OFED v3.4 or later is installed.

     

    Configuration

    1. Check the default RoCE mode for RDMA CM applications.

    # cma_roce_mode -d mlx5_0 -p 1

    IB/RoCE V1

     

    If you capture traffic at this point, you will only see RoCE v1 traffic (no IP header).

     

     

    2. Set the default RoCE mode for RDMA CM applications. Use -m 2 parameter.

    # cma_roce_mode -d mlx5_0 -p 1 -m 2

    RoCE V2

     

    Verification

    When using perftest package to send traffic, use the -R flag to force RDMA CM.

     

    For example:

    Run on the server:

    # ib_write_bw --report_gbits -D5 -F -R

    And on the client (the IP is 1.1.1.2):

    # ib_write_bw --report_gbits -D5 -F -R 1.1.1.2

     

    In case you capture the packets on the switch, you will see the UDP header on the RDMA CM traffic.

     

    Note: If -R flag will not be used, the configuration of cma_roce_mode will have no effect.

    In this case, RoCE v1.5 (RoCE over IP, no UDP).

    # ib_write_bw --report_gbits -D5 -F