HowTo Set Egress Priority VLAN on Linux

Version 18

    This post shows basic configuration example on how to set the egress priority of a VLAN interface.

    This post is basic and is meant for beginners.

     

    References

     

    Setup

    • Two hosts connected via a switch or back to back. This example was given based on servers with CentoOS 7.1, other distributions should be similar.
    • In case using a switch, Make sure that the VLAN is enabled on the relevant switch ports.

     

    Configuration

    Note: This procedure is relevant for traffic that passes via the kernel, and not for RoCE traffic (bypass).

     

    1. Enable 802.1q on the server.

    # modprobe 8021q

     

    2. Assuming you wish to create a VLAN 100 under the physical port ens785f1, Create the file /etc/sysconfig/network-scripts/ifcfg-ens785f1.100 with the following attributes

    VLAN=yes

    TYPE=Vlan

    DEVICE=ens785f1.100

    PHYSDEV=ens785f1

    VLAN_ID=100

    REORDER_HDR=0

    VLAN_EGRESS_PRIORITY_MAP=0:3

    # Another options is to map more priorities. For example, in this case all kernel priorities are mapped to egress priority 3.

    # VLAN_EGRESS_PRIORITY_MAP=0:3,1:3,2:3,3:3,4:3,5:3,6:3,7:3

    BOOTPROTO=static

    DEFROUTE=yes

    IPV4_FAILURE_FATAL=no

    NAME=ens785f1.100

    ONBOOT=yes

    IPADDR=12.12.12.5

    NETMASK=255.255.255.0

    NM_CONTROLLED=no

     

    Note: The VLAN_EGRESS_PRIORITY_MAP parameter maps the Kernel SKB priority to the egress priority. in this example skb priority 0 is mapped to VLAN egress priority 3.

     

    3. Restart the network services

    # service network restart

     

    4. Check the VLAN configuration

     

    This is the output when the VLAN egress priority mapping is as follows:

    • VLAN_EGRESS_PRIORITY_MAP=0:3,1:3,2:3,3:3,4:3,5:3,6:3,7:3

    # cat /proc/net/vlan/ens785f1.100

    ens785f1.100  VID: 100 REORDER_HDR: 1  dev->priv_flags: 1

             total frames received            0

              total bytes received            0

          Broadcast/Multicast Rcvd            0

     

     

          total frames transmitted           18

           total bytes transmitted         3310

    Device: ens785f1

    INGRESS priority mappings: 0:0  1:0  2:0  3:0  4:0  5:0  6:0 7:0

    EGRESS priority mappings: 0:3 1:3 2:3 3:3 4:3 5:3 6:3 7:3

     

    5. For Ethernet only (no RDMA/RoCE) in case of ConnectX-3/Pro only, run the following command to map all the traffic to specific egress priority.

    # echo "ens785f1.100 3" > /sys/fs/cgroup/net_prio/net_prio.ifpriomap

    Note: This is not needed for ConnectX-4.

    For more info, see: net_prio in REHL and net_prio in kernel.

     

    6. Check the traffic between the hosts.

         Once you try to ping to the remote host, you will see the priority bit equal to 3.

    1.PNG

     

     

    Troubleshooting

    1. In cases the VLAN_EGRESS_PRIORITY_MAP=...  doesn't work when adding it to the ifcfg-eth1 file. use ip link command as follows:

     

    # ip link set ens785f1.100 vlan type egress 0:4

    # cat /proc/net/vlan/ens785f1.100

    ens785f1.100  VID: 100 REORDER_HDR: 0  dev->priv_flags: 1

             total frames received            0

              total bytes received            0

          Broadcast/Multicast Rcvd            0

     

     

          total frames transmitted           23

           total bytes transmitted         3601

    Device: enp5s0

    INGRESS priority mappings: 0:0  1:0  2:0  3:0  4:0  5:0  6:0 7:0

    EGRESS priority mappings: 0:4

     

    2. In other cases, you can install the vconfig package, and use the vconfig set_egress_map command.

    # yum install vconfig

    See HowTo Run RoCE over L2 Enabled with PFC

     

    For example:

    # for i in {0..7}; do vconfig set_egress_map eth1.100 $i 3 ; done

    Set egress mapping on device -:eth1.100:- Should be visible in /proc/net/vlan/eth1.100

    Set egress mapping on device -:eth1.100:- Should be visible in /proc/net/vlan/eth1.100

    Set egress mapping on device -:eth1.100:- Should be visible in /proc/net/vlan/eth1.100

    Set egress mapping on device -:eth1.100:- Should be visible in /proc/net/vlan/eth1.100

    Set egress mapping on device -:eth1.100:- Should be visible in /proc/net/vlan/eth1.100

    Set egress mapping on device -:eth1.100:- Should be visible in /proc/net/vlan/eth1.100

    Set egress mapping on device -:eth1.100:- Should be visible in /proc/net/vlan/eth1.100

    Set egress mapping on device -:eth1.100:- Should be visible in /proc/net/vlan/eth1.100

    #

     

    Egress Mapping for RoCE Traffic

    For RoCE traffic you need to use the tc_wrap command, see examples here: