HowTo Create Linux bond (LAG) Interface over InfiniBand network

Version 11

    This post is a basic post and made for beginners that wishes to enable bond interface over InfiniBand network.

     

    References

     

    Setup

    1%3Fauth_token%3Df9cc280bd6233e91451735df3c68f7cb78b83a49

    Prerequisites

     

    Configuration

     

     

    1. Create bond0 Interface on the server. Update the following file:

     

    /etc/sysconfig/network-scripts/ifcfg-bond0

     

    Note: the only meaningful bonding policy in IPoIB is High-Availability (bonding mode number 1, or "active-backup").

    To learn more about the Linux bonding options click here.

     

    Here is an example:

    DEVICE=bond0

    IPADDR=11.11.11.1

    NETWORK=11.11.11.0

    NETMASK=255.255.255.0

    BROADCAST=11.11.11.255

    USERCTL=no

    BOOTPROTO=none

    ONBOOT=yes

    NM_CONTROLLED=yes

    BONDING_OPTS="mode=active-backup primary=ib0 miimon=100 updelay=100 downdelay=100"

    MTU=2044

     

    2. Update/Create ib0 Interface on the server. Update the following file:

     

    /etc/sysconfig/network-scripts/ifcfg-ib0

     

    Here is an example:

    DEVICE=ib0

    TYPE=InfiniBand

    NM_CONTROLLED=yes

    ONBOOT=yes

    USERCTL=no

    MASTER=bond0

    SLAVE=yes

    BOOTPROTO=none

    PRIMARY=yes

     

    3. Update/Create ib1 Interface on the server. Update the following file:

     

    /etc/sysconfig/network-scripts/ifcfg-ib1

     

    Here is an example:

    DEVICE=ib1

    TYPE=InfiniBand

    NM_CONTROLLED=yes

    ONBOOT=yes

    USERCTL=no

    MASTER=bond0

    SLAVE=yes

    BOOTPROTO=none

    PRIMARY=no

     

    4. Create/Update the bond.conf file.

     

    /etc/modprobe.d/bond.conf

     

    Configure this file as follows:

    alias bond0 bonding                             # Adapter bonding driver

    options bond0 max_bonds=2 miimon=100 mode=1     # 100ms fail-over timer. Mode 1 = Active/Backup

     

    5. Restart the network services and the driver.

    # /etc/init.d/network restart

    ...

    # /etc/init.d/openibd restart

    ...

     

    6. Verify the bond configuration:

    # cat /proc/net/bonding/bond0

    Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

     

    Bonding Mode: fault-tolerance (active-backup) (fail_over_mac active)

    Primary Slave: ib0 (primary_reselect always)

    Currently Active Slave: ib0

    MII Status: up

    MII Polling Interval (ms): 100

    Up Delay (ms): 100

    Down Delay (ms): 100

     

    Slave Interface: ib1

    MII Status: up

    Speed: 40000 Mbps

    Duplex: full

    Link Failure Count: 0

    Permanent HW addr: a0:04:03:00:fe:80

    Slave queue ID: 0

     

    Slave Interface: ib0

    MII Status: up

    Speed: 40000 Mbps

    Duplex: full

    Link Failure Count: 254

    Permanent HW addr: a0:04:02:20:fe:80

    Slave queue ID: 0

     

    Fail Over MAC

    Since rdma_cm does not support MAC enslavement, fail_over_mac=1 (or active) should be added to the bond interface configuration in case of Ethernet interface.

     

    The value active(1) for fail_over_mac policy indicates that the MAC address of the bond should always be the MAC address of the currently active slave. The MAC address of the slaves is not changed; instead, the MAC address of the bond changes during a failover.

    Changing to fail_over_mac=1 solves this problem as if this not active, the bond MAC jumps from one interface to another according to the active interface.

     

    Add the bonding options (BONDING_OPT)  "fail_over_mac=1" to /etc/sysconfig/network-scripts/ifcfg-bond0.

     

    Note: the default of this parameter is disabled (0).

     

    Here is an example:

    DEVICE=bond0

    IPADDR=11.11.11.1

    NETWORK=11.11.11.0

    NETMASK=255.255.255.0

    BROADCAST=11.11.11.255

    USERCTL=no

    BOOTPROTO=none

    ONBOOT=yes

    NM_CONTROLLED=yes

    BONDING_OPTS="mode=active-backup primary=ib0 miimon=100 updelay=100 downdelay=100 fail_over_mac=1"

    MTU=2044