This post discusses and shows how to configure LAG (bonding) in the Hypervisor, while the VM interface is configured with SR-IOV over Mellanox ConnectX-3 Pro adapter card (Ethernet).
It is possible to configure bonding over two interfaces in the Hypervisor in case the Hypervisor has two active ports.
Note: This is supported currently for Ethernet traffic only. RMDA/RoCE support will be added in the future.
- HowTo Map VF Number and the VM in SR-IOV Mode
- HowTo Set Virtual Network Attributes on a Virtual Function (SR-IOV)
- Link Aggregation and LACP with Open vSwitch
- HowTo Configure SR-IOV VFs on Different ConnectX-3 Ports
How Does It Work
When bonding mode is set to 1 (active-backup), all traffic will be directed to the active port as the bonding module shows (see /proc/net/bonding/bond0). When this port fails, all traffic from all VFs will be directed to the other port.
When bonding mode is set to 4 (active-active), traffic from each VF will be directed to the port it is defined for, as long as this port is up (the two ports are sending traffic).
- Linux kernel 4.5
- Inbox drivers or MLNX_OFED 3.4.1
The solution is limited to configuration where only single port VFs exist. VFs can be spread among both ports for maximal utilization of the hardware capacity. However, a single VF is still limited to one port's throughput. In the Hypervisor, a bonding device should be configured to enslave the 2 netdev interfaces of the adapter in bond mode 1 (active-backup) or 4 (dynamic active-active).
Note: MLNX_OFED is not required in this case.
1. Configure SR-IOV with single port VF.
2. Remove PFs from OVS.
3. Configure bonding.
4. Add bind interface to OVS bridge.
Example of bonding on Fedora24 over devstack:
1. Install devstack.
2. Remove PFs from OVS:
3. Add ports to bond:
# ifconfig p1 down; ifconfig p2 down
# modprobe bonding mode=1 miimon=100
# echo +p1 > /sys/class/net/bond0/bonding/slaves
# echo +p2 > /sys/class/net/bond0/bonding/slaves
# ifconfig bond0 up
4. Add bond0 interface to OVS:
# sudo ovs-vsctl add-port br-p1 bond0
To remove the bond (unbond), run:
# echo -p1 > /sys/class/net/bond0/bonding/slaves
# echo -p2 > /sys/class/net/bond0/bonding/slaves
# modprobe -r bonding
- If there is no communication between VMs on different Hypervisors, check (or disable) iptables.service.
- OVS bonding is not supported, see Link Aggregation and LACP with Open vSwitch · Scott's Weblog · The weblog of an IT pro specializing in virtualization.
- Bonding supported with single port VFs only, see HowTo Configure SR-IOV VFs on Different ConnectX-3 Ports.