HowTo Configure LAG over VF (ConnectX-3 Pro)

Version 18

    This post discusses and shows how to configure LAG (bonding) in the Hypervisor, while the VM interface is configured with SR-IOV over Mellanox ConnectX-3 Pro adapter card (Ethernet).

    It is possible to configure bonding over two interfaces in the Hypervisor in case the Hypervisor has two active ports.

    Note: This is supported currently for Ethernet traffic only. RMDA/RoCE support will be added in the future.

     

    References

     

    How Does It Work

    When bonding mode is set to 1 (active-backup), all traffic will be directed to the active port as the bonding module shows (see /proc/net/bonding/bond0). When this port fails, all traffic from all VFs will be directed to the other port.

    When bonding mode is set to 4 (active-active), traffic from each VF will be directed to the port it is defined for, as long as this port is up (the two ports are sending traffic).

     

    Prerequisites

    • Linux kernel 4.5
    • Inbox drivers or MLNX_OFED 3.4.1

     

    Overview

    The solution is limited to configuration where only single port VFs exist. VFs can be spread among both ports for maximal utilization of the hardware capacity. However, a single VF is still limited to one port's throughput. In the Hypervisor, a bonding device should be configured to enslave the 2 netdev interfaces of the adapter in bond mode 1 (active-backup) or 4 (dynamic active-active).

     

     

    Note: MLNX_OFED is not required in this case.

     

    Host Configuration

    1. Configure SR-IOV with single port VF.

    2. Remove PFs from OVS.

    3. Configure bonding.

    4. Add bind interface to OVS bridge.

     

    Example of bonding on Fedora24 over devstack:

     

    1. Install devstack.

     

    2. Remove PFs from OVS:

     

    3. Add ports to bond:

    # ifconfig p1 down; ifconfig p2 down

    # modprobe bonding mode=1 miimon=100

    # echo +p1 > /sys/class/net/bond0/bonding/slaves

    # echo +p2 > /sys/class/net/bond0/bonding/slaves

    # ifconfig bond0 up

     

    4. Add bond0 interface to OVS:

    # sudo ovs-vsctl add-port br-p1 bond0

     

    To remove the bond (unbond), run:

        # echo -p1 > /sys/class/net/bond0/bonding/slaves

        # echo -p2 > /sys/class/net/bond0/bonding/slaves

        # modprobe -r bonding

     

    Known issues