HowTo Configure MLAG + Gateway HA

Version 11

    This post shows how to configure two Ethernet switches in MLAG and two Gateways in HA mode.

    This post is meant for IT managers and assumes basic understanding in Mellanox switch configuration, InfiniBand and networking in general.

     

    >>Learn how to configure MLAG for free on the Mellanox Academy

     

    References

     

    Setup

     

    In this example, S1 and S2 are serves equipped with ConnectX-3 installed with CentOS 7, while sx01, sx02, sx03 and sx04 are Mellanox Gateways SX6036G.

     

    2.png

     

    Important Note: This setup is focus on the connectivity of the Gateway HA towards the MLAG Domain. The links between the sx01 and sx02 switches is required just because there is no other InfiniBand switch connected to them in the network. This link is not needed if the server S2 and the switches sx01 and sx02 are part of a larger InfiniBand subnet.

     

    Configuration

    1. Configure switches sx03 and sx04 as MLAG cluster, refer to HowTo Configure MLAG on Mellanox Switches

     

    2. Configure switches sx01 and sx02 as Gateway HA cluster, refer to HowTo Configure InfiniBand Gateway HA (Proxy ARP)

     

    3. Configure the links between the MLAG cluster and the Gateway HA cluster:

    • On sx03 and sx04, create an mlag-port-channel 1 towards sx01 that consist of 2 interfaces each. on sx01 configure those ports as port-channel.
    • On sx03 and sx04, create an mlag-port-channel 2 towards sx02 that consist of 2 interfaces each. on sx02 configure those ports as port-channel.

     

    Note: you can configure LACP or static LAG on those links, in this example we used LACP ('mode active') for example:

     

    switch (config) # interface ethernet 1/7 mlag-channel-group 1 mode active

     

    4. Configure bond0 on S1 server as static or LACP bond. For S2 bond over InfiniBand, refer to HowTo Create Linux bond (LAG) Interface over InfiniBand network.

     

    Switch Running-Config Output Example

    In the below output, you can find the running configuration of the four switches. The port connectivity and enhancements are mentioned in red in the output.

     

    sx01

     

    ##

    ## Port configuration

    ##

       port 1/1 type infiniband force -> this port is connected to the server S2

       port 1/2 type infiniband force -> this port is connected to the server sx02

       port 1/3 type infiniband force -> this port is connected to the server sx02

     

    ##

    ## Interface Ethernet configuration

    ##

       interface port-channel 1 -> this is the port channel connected to the MLAG switch cluster (one port channel that have four members)

       interface ethernet 1/7 channel-group 1 mode active -> all LAG members are LACP

       interface ethernet 1/8 channel-group 1 mode active -> all LAG members are LACP

       interface ethernet 1/9 channel-group 1 mode active -> all LAG members are LACP

       interface ethernet 1/10 channel-group 1 mode active -> all LAG members are LACP

       interface port-channel 1 switchport mode trunk

     

    ##

    ## VLAN configuration

    ##

       vlan 50

       interface port-channel 1 switchport trunk allowed-vlan none

       interface port-channel 1 switchport trunk allowed-vlan 50

     

    ##

    ## LLDP configuration

    ##

       lldp

     

    ##

    ## Proxy Arp configuration

    ##

       interface proxy-arp 1

       interface proxy-arp 1 ha member sx01

       interface proxy-arp 1 ha member sx01 ip address 20.20.20.2

       interface proxy-arp 1 ha member sx02

       interface proxy-arp 1 ha member sx02 ip address 20.20.20.3

       interface proxy-arp 1 ip netmask /24

       interface proxy-arp 1 ip pkey 0x7FFF

       interface proxy-arp 1 ip vlan 50

    no interface proxy-arp 1 shutdown

       ip proxy-arp

       proxy-arp ha GW-HA-Group ip 10.20.2.190 /24 force

     

     

    sx02

     

    ##

    ## Port configuration

    ##

       port 1/1 type infiniband force -> this port is connected to the server S2

       port 1/2 type infiniband force -> this port is connected to the server sx01

       port 1/3 type infiniband force -> this port is connected to the server sx01

     

     

    ##

    ## Interface Ethernet configuration

    ##

       interface port-channel 1 -> this is the port channel connected to the MLAG switch cluster (one port channel that have four members)

       interface ethernet 1/7 channel-group 1 mode active -> all LAG members are LACP

       interface ethernet 1/8 channel-group 1 mode active -> all LAG members are LACP

       interface ethernet 1/9 channel-group 1 mode active -> all LAG members are LACP

       interface ethernet 1/10 channel-group 1 mode active -> all LAG members are LACP

       interface port-channel 1 switchport mode trunk

     

     

    ##

    ## VLAN configuration

    ##

       vlan 50

       interface port-channel 1 switchport trunk allowed-vlan none

       interface port-channel 1 switchport trunk allowed-vlan 50

     

    ##

    ## LLDP configuration

    ##

       lldp

     

    ##

    ## Proxy Arp configuration

    ##

       interface proxy-arp 1

       interface proxy-arp 1 ha member sx01

       interface proxy-arp 1 ha member sx01 ip address 20.20.20.2

       interface proxy-arp 1 ha member sx02

       interface proxy-arp 1 ha member sx02 ip address 20.20.20.3

       interface proxy-arp 1 ip netmask /24

       interface proxy-arp 1 ip pkey 0x7FFF

       interface proxy-arp 1 ip vlan 50

    no interface proxy-arp 1 shutdown

       ip proxy-arp

       proxy-arp ha GW-HA-Group ip 10.20.2.190 /24 force

     

     

    sx03

     

    ##

    ## MLAG protocol

    ##

       protocol mlag

     

    ##

    ## DCBX PFC configuration

    ##

       dcb priority-flow-control enable force

       interface ethernet 1/35 dcb priority-flow-control mode on force -> IPL member port

       interface ethernet 1/36 dcb priority-flow-control mode on force -> IPL member port

       interface port-channel 1 dcb priority-flow-control mode on force -> IPL port-channel

     

    ##

    ## Interface Ethernet configuration

    ##

       interface mlag-port-channel 1 -> This MLAG port channel is connected to the sx01 (MPo1), two members

       interface mlag-port-channel 2 -> This MLAG port channel is connected to the sx02 (MPo2), two members

       interface mlag-port-channel 3 -> This MLAG port channel is connected to the S1 (MPo3), one member

       interface port-channel 1 -> IPL

       interface ethernet 1/1 mlag-channel-group 3 mode on

       interface ethernet 1/7 mlag-channel-group 1 mode active

       interface ethernet 1/8 mlag-channel-group 1 mode active

       interface ethernet 1/9 mlag-channel-group 2 mode active

       interface ethernet 1/10 mlag-channel-group 2 mode active

       interface ethernet 1/35 channel-group 1 mode active  -> IPL

       interface ethernet 1/36 channel-group 1 mode active  -> IPL

      interface mlag-port-channel 1 switchport mode trunk

       interface mlag-port-channel 2 switchport mode trunk

       interface mlag-port-channel 1 no shutdown

       interface mlag-port-channel 2 no shutdown

       interface mlag-port-channel 3 no shutdown

     

    ##

    ## VLAN configuration

    ##

       vlan 50

       vlan 4001  -> IPL VLAN

       interface mlag-port-channel 3 switchport access vlan 50 -> -> This MLAG port channel is connected to the S1 (MPo3)

     

    ##

    ## STP configuration

    ##

    no spanning-tree

     

     

    ##

    ## LAG configuration

    ##

       lacp

     

    ##

    ## LLDP configuration

    ##

       lldp

     

    ##

    ## L3 configuration

    ##

       ip routing

       interface vlan 4001 -> Used for IPL

       interface vlan 4001 ip address 10.10.10.1 255.255.255.0 -> Used for IPL

     

    ##

    ## MLAG configurations

    ##

       mlag-vip MLAG-Group ip 10.20.2.191 /24 force

    no mlag shutdown

       interface port-channel 1 ipl 1

       interface vlan 4001 ipl 1 peer-address 10.10.10.2

     

     

    sx04

     

    ##

    ## MLAG protocol

    ##

       protocol mlag

     

    ##

    ## DCBX PFC configuration

    ##

       dcb priority-flow-control enable force 

       interface ethernet 1/35 dcb priority-flow-control mode on force  -> IPL member port

       interface ethernet 1/36 dcb priority-flow-control mode on force  -> IPL member port

       interface port-channel 1 dcb priority-flow-control mode on force -> IPL  port channel

     

    ##

    ## Interface Ethernet configuration

    ##

       interface mlag-port-channel 1 -> This MLAG port channel is connected to the sx01 (MPo1), two members

       interface mlag-port-channel 2 -> This MLAG port channel is connected to the sx02 (MPo2), two members

       interface mlag-port-channel 3 -> This MLAG port channel is connected to the S1 (MPo3), one member

       interface port-channel 1 -> IPL

       interface ethernet 1/1 mlag-channel-group 3 mode on

       interface ethernet 1/7 mlag-channel-group 2 mode active

       interface ethernet 1/8 mlag-channel-group 2 mode active

       interface ethernet 1/9 mlag-channel-group 1 mode active

       interface ethernet 1/10 mlag-channel-group 1 mode active

       interface ethernet 1/35 channel-group 1 mode active

       interface ethernet 1/36 channel-group 1 mode active

       interface mlag-port-channel 1 switchport mode trunk

       interface mlag-port-channel 2 switchport mode trunk

       interface mlag-port-channel 1 no shutdown

       interface mlag-port-channel 2 no shutdown

       interface mlag-port-channel 3 no shutdown

     

    ##

    ## VLAN configuration

    ##

       vlan 50

       vlan 4001 -> IPL VLAN

       interface mlag-port-channel 3 switchport access vlan 50 -> Used for IPL

     

    ##

    ## STP configuration

    ##

    no spanning-tree

     

    ##

    ## LAG configuration

    ##

       lacp

     

    ##

    ## LLDP configuration

    ##

       lldp

     

    ##

    ## L3 configuration

    ##

       ip routing

       interface vlan 4001 -> Used for IPL

       interface vlan 4001 ip address 10.10.10.2 255.255.255.0 -> Used for IPL

     

    ##

    ## MLAG configurations

    ##

       mlag-vip MLAG-Group ip 10.20.2.191 /24 force

    no mlag shutdown

       interface port-channel 1 ipl 1

       interface vlan 4001 ipl 1 peer-address 10.10.10.1