HowTo Set the Default RoCE Mode When Using RDMA CM

Version 4

    This post describes how to change the default configuration of RoCE mode for RDMA CM applications using the cma_roce_mode tool.

    Note: This is relevant for MLNX_OFED v3.4 and above.




    Setup Example

    • Two servers equipped with ConnectX-4 adapters connected to a Spectrum switch.
    • Additional server can be connected to Spectrum for monitoring (e.g. install wireshark and capture the traffic).
    • MLNX_OFED v3.4 or later is installed.



    1. Check the default RoCE mode for RDMA CM applications.

    # cma_roce_mode -d mlx5_0 -p 1

    IB/RoCE V1


    If you capture traffic at this point, you will only see RoCE v1 traffic (no IP header).



    2. Set the default RoCE mode for RDMA CM applications. Use -m 2 parameter.

    # cma_roce_mode -d mlx5_0 -p 1 -m 2

    RoCE V2



    When using perftest package to send traffic, use the -R flag to force RDMA CM.


    For example:

    Run on the server:

    # ib_write_bw --report_gbits -D5 -F -R

    And on the client (the IP is

    # ib_write_bw --report_gbits -D5 -F -R


    In case you capture the packets on the switch, you will see the UDP header on the RDMA CM traffic.


    Note: If -R flag will not be used, the configuration of cma_roce_mode will have no effect.

    In this case, RoCE v1.5 (RoCE over IP, no UDP).

    # ib_write_bw --report_gbits -D5 -F