This post shows the command to configure ToS/DSCP for RDMA-CM QPs only.
This feature is supported starting from MLNX_OFED 4.0 and onward, and supported for ConnectX-4 adapters.
A new entry was added to the RDMA-CM configfs that allows users to select default TOS for RDMA-CM QPs. This is useful for users that want to control the TOS field without changing their code. Other applications that set the TOS explicitly using the rdma_set_option API will continue to work as expected overriding the configfs value.
In order to configure the TOS, the users can use the cma_roce_tos script that is part of MLNX_OFED 4.0
# cma_roce_tos –d <ib_device> -t <tos>
Note: The ToS field is 8 bits, while the DSCP is 6 bits. To set a DSCP value of X, you need to multiply this value by 4 (SHIFT 2). For example, to set DSCP value of 24, (24x4=96). Set the ToS bit to 96.
Note: In case you run perftest package (e.g. ib_send_bw), running this command will change the default for the -T option (TOS default). In other words, no need to run -T <tos> if this command is used.
Note: ECN configuration is done differently, via the sysfs or mlxconfig, see: HowTo Configure DCQCN (RoCE CC) for ConnectX-4 (Linux).
Note: make sure to change the default RDMA CM mode to RoCE v2, See HowTo Set the Default RoCE Mode When Using RDMA CM.
Note: The command is not persistent.