RDMA CM and RoCE Version Defaults

Version 10

    RDMA CM is run by default over RoCEv1. With MLNX_OFED 4.1, the default changed to RoCEv2.

    This change is submitted for kernel 4.12.

     

    References

     

    Note: This default does not apply for ConnectX-3 or older adapter cards which support only RoCEv1.

     

    The following defaults are applied:

     

    RDMA CM Default for RoCEv1 (Type 1):

    • ConnectX-3 or older adapter cards (HW does not support)
    • When using MLNX_OFED 4.0 or older.

     

    RDMA CM Default to RoCEv2 (Type 2):

    • When using ConnectX-3 Pro, ConnectX-4, ConnectX-4 Lx and ConnectX-5 with MLNX_OFED 4.1

     

    RDMA_CM session require that both sides support the same RoCE mode. Since the mode of the session is determined by active side (client), the below table for out-of-the-box behavior is provided.

     

    Client ServerOut of the Box
    Type 1Type 1OK
    Type 1Type 2OK
    Type 2Type 2OK
    Type 2Type 1No Connection

     

    To solve this issue, please perform one of the following:

    1. For adapter cards that support RoCEv2 (ConnectX-3 or later), consider upgrading the hosts to MLNX_OFED 4.1, or change the default RoCE mode (see HowTo Set the Default RoCE Mode When Using RDMA CM).

        This leads RDMA_CM applications to work by default (RoCE v2).

     

    2. If upgrade is impossible or when the NIC's server does not support RoCEv2, change the default RoCE mode in RDMA_CM to RoCEv1 (see HowTo Set the Default RoCE Mode When Using RDMA CM).