HowTo Set Egress ToS/DSCP on RDMA-CM QPs

Version 5

    This post shows the command to configure ToS/DSCP for RDMA-CM QPs only.

    This feature is supported starting from MLNX_OFED 4.0 and onward, and supported for ConnectX-4 adapters.




    A new entry was added to the RDMA-CM configfs that allows users to select default TOS for RDMA-CM QPs. This is useful for users that want to control the TOS field without changing their code. Other applications that set the TOS explicitly using the rdma_set_option API will continue to work as expected overriding the configfs value.


    In order to configure the TOS, the users can use the cma_roce_tos script that is part of MLNX_OFED 4.0


    Usage example:

    # cma_roce_tos –d <ib_device> -t <tos>

    Note: The ToS field is 8 bits, while the DSCP is 6 bits. To set a DSCP value of X, you need to multiply this value by 4 (SHIFT 2). For example, to set DSCP value of 24, (24x4=96). Set the ToS bit to 96.


    Note: In case you run perftest package (e.g. ib_send_bw), running this command will change the default for the -T option  (TOS default). In other words, no need to run -T <tos> if this command is used.


    Note: ECN configuration is done differently, via the sysfs or mlxconfig, see: HowTo Configure DCQCN (RoCE CC) for ConnectX-4 (Linux).


    Note: make sure to change the default RDMA CM mode to RoCE v2, See HowTo Set the Default RoCE Mode When Using RDMA CM.


    Note: The command is not persistent.