RDMA CM is run by default over RoCEv1. With MLNX_OFED 4.1, the default changed to RoCEv2.
This change is submitted for kernel 4.12.
Note: This default does not apply for ConnectX-3 or older adapter cards which support only RoCEv1.
The following defaults are applied:
RDMA CM Default for RoCEv1 (Type 1):
- ConnectX-3 or older adapter cards (HW does not support)
- When using MLNX_OFED 4.0 or older.
RDMA CM Default to RoCEv2 (Type 2):
- When using ConnectX-3 Pro, ConnectX-4, ConnectX-4 Lx and ConnectX-5 with MLNX_OFED 4.1
RDMA_CM session require that both sides support the same RoCE mode. Since the mode of the session is determined by active side (client), the below table for out-of-the-box behavior is provided.
|Client||Server||Out of the Box|
|Type 1||Type 1||OK|
|Type 1||Type 2||OK|
|Type 2||Type 2||OK|
|Type 2||Type 1||No Connection|
To solve this issue, please perform one of the following:
1. For adapter cards that support RoCEv2 (ConnectX-3 or later), consider upgrading the hosts to MLNX_OFED 4.1, or change the default RoCE mode (see HowTo Set the Default RoCE Mode When Using RDMA CM).
This leads RDMA_CM applications to work by default (RoCE v2).
2. If upgrade is impossible or when the NIC's server does not support RoCEv2, change the default RoCE mode in RDMA_CM to RoCEv1 (see HowTo Set the Default RoCE Mode When Using RDMA CM).