HowTo Configure QinQ Encapsulation per VF in Linux (VST) for ConnectX-3 Pro

Version 20

    The goal of this post is to present QinQ VLAN tag (S-VLAN) configuration to the hypervisor per VF. This solution is supported in ConnectX-3 Pro adapters ONLY (but not supported in ConnectX-4 adapters yet), starting with Mellanox OFED 3.3.2.

     

    References

     

    Overview

    In this case, the VM attached to the VF (SR-IOV) can send traffic with or without C-VLAN.

    Once a VF is configured to VST QinQ encapsulation (VST QinQ), the adapter's hardware will insert S-VLAN to any packet from the VF to physical port. On the receive side, the adapter hardware will strip the S-VLAN from any packet coming from the wire to that VF.

     

    Hair-pin Topology

    "Hair-pin" topology is supported for this feature. Hair-pin topology means that two VMs will be able to communicate (e.g. ping) between them over the same C-VLAN once each one of them is configured with the same S-VLAN over different VFs on the same port. In the figure below, both VMs could communicate over C-VLAN 40 once configured over S-VLAN 100.

    Network Considerations

    The network switches may require increasing the MTU on the relevant switch ports.

     

    Setup

    The setup assumes two servers equipped with ConnectX-3 Pro adapters.

     

     

     

    Prerequisite

    Follow HowTo Configure SR-IOV for ConnectX-3 with KVM (Ethernet)  in order to configure the server in SR-IOV virtualization environment.

     

    Configuration

    1. Enable QinQ support and set phv-bit flag using ethtool (on the hypervisor).

    # ethtool --set-priv-flags ens2 phv-bit on

     

    2. Add the required S-VLAN (QinQ) tag (on the hypervisor) per port and VF.

     

    There are two available options:

    Option 1: using sysfs - in cases of using kernel version older than 4.9:

    # echo 'vlan 100 proto 802.1ad' > /sys/class/net/ens2/vf0/vlan_info

    To view the configuration, cat the sysfs vlan_info field as follows:

    # cat /sys/class/net/ens2/vf0/vlan_info

    vlan 100, vlan protocol 802.1ad

     

    Option 2: using the "ip link" command (latest version), kernel version 4.9 or above.

    # ip link set dev ens2 vf 0 vlan 100 proto 802.1ad

     

    Check the configuration using ip link show command:

    # ip link show ens2

    2: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

        link/ether 7c:fe:90:19:9e:21 brd ff:ff:ff:ff:ff:ff

        vf 0 MAC 00:00:00:00:00:00, vlan 100, vlan protocol 802.1ad , spoof checking off, link-state auto

        vf 1 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto

        vf 2 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto

        vf 3 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto

        vf 4 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto

     

    4. (Optional) Add S-VLAN priority. Use qos parameter in ip link command (or sysfs).

     

    # ip link set dev ens2 vf 0 vlan 100 qos 3 proto 802.1ad

     

    Check the configuration using ip link show command:

     

    # ip link show ens2

    2: ens2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

        link/ether 7c:fe:90:19:9e:21 brd ff:ff:ff:ff:ff:ff

        vf 0 MAC 00:00:00:00:00:00, vlan 100, qos 3, vlan protocol 802.1ad , spoof checking off, link-state auto

        vf 1 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto

        vf 2 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto

        vf 3 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto

        vf 4 MAC 00:00:00:00:00:00, vlan 4095, spoof checking off, link-state auto

     

    5. Restart the driver in the VM attached to that VF:

    (VM1)# /etc/init.d/openidb restart

     

    6. Create VLAN interface on the VM and add IP address.

    # ip link add link ens5 ens5.40 type vlan protocol 802.1q id 40

    # ip addr add 42.134.135.7/16 brd 42.134.255.255 dev ens5.40

    # ip link set dev ens5.40 up

     

    Verification

    One way to test this feature is to use port mirroring on the network switch and capture the packet being sent from server A to server B to external server C.

    In case you are using Mellanox switches, refer to HowTo Configure Port Mirroring on Mellanox Ethernet Switches.

    Run ping between the VMs and open wireshark or use tcpdump to capture the packet.

    See the attached wireshark pcap file output example.

     

    Notes:

    1. It is not possible to capture the packets on Server A or B on the hypervisor, via wireshark or tcpdump as the traffic doesn't pass via the kernel (SR-IOV).

    2. If you capture the packets on the VM, the S-VLAN will not be visible there. The VM is not aware of it.