Mellanox MLAG and Cisco vPC Configuration and Troubleshooting

Version 5

    This post describes a setup that includes Mellanox MLAG switches connected to Cisco Nexus vPC. Due to the fact that Mellanox MLAG features do not support spanning tree with MLAG, this leads to potential network loops which would cause a network down if the connection process or configuration is not correct.

     

     

    Setup

    Below is the setup used. SX03 and SX04 are Mellanox Ethernet switches while Nexus1 and Nexus2 are Cisco Nexus switches. S3 and S4 are two servers equipped with dual port ConnectX®-3 Pro adapters.

     

    Configuration

     

    Note: The thing to worry about here is networks loops. Therefore, firstly run all the configuration needed, and only then connect the cables. In case all cables are already connected, make sure to leave the mlag-port-channel in admin down state until you finish the configuration setup to avoid loops.

     

     

    Cisco Nexus Switch Configuration

    On the Cisco side, below is the running-config for vPC.

    Port-channel 1 (Ethernet 2/3 and 2/4) is for vpc peer-link.

    Port-channel 100 (Ethernet 1/1 and 1/2) is used to connect Mellanox switches.

    feature interface-vlan

    feature lacp

    feature vpc

    vlan 1

    vrf context management

      ip route 0.0.0.0/0 10.20.0.251

    vpc domain 1

      peer-switch

      role priority 1000

      peer-keepalive destination 10.20.2.15 source 10.20.2.14

      delay restore 150

    interface Vlan1

      no shutdown

      ip address 192.168.1.1/24

    interface port-channel1

      spanning-tree port type network

      speed 40000

      vpc peer-link

    interface port-channel100

      speed 10000

      vpc 100

    interface port-channel200

      speed 10000

      vpc 200

    interface Ethernet1/1

      channel-group 100 mode active

    interface Ethernet1/2

      channel-group 100 mode active

    interface Ethernet2/3

      channel-group 1 mode active

    interface Ethernet2/4

      channel-group 1 mode active

     

    Mellanox MLAG configuration

    Port-channel 1 (Ethernet 1/35 and 1/36) is an IPL link.

    Mlag-port-channel 100 (Ethernet 1/1 and 1/2) connects to the Cisco Nexus vPC pair.

    Please follow the standard step to configure MLAG (described in HowTo Configure MLAG on Mellanox Switches).

    Make sure that mlag-port-channel 100 is shut down before connecting it to the Cisco vPC pair.

     

    protocol mlag

       interface ethernet 1/1 speed 10000 force

       interface ethernet 1/2 speed 10000 force

       interface mlag-port-channel 100

       interface port-channel 1

       interface ethernet 1/1 mlag-channel-group 100 mode active

       interface ethernet 1/2 mlag-channel-group 100 mode active

       interface ethernet 1/35 channel-group 1 mode active

       interface ethernet 1/36 channel-group 1 mode active

       vlan 4000

    no spanning-tree

       lacp

       ip routing vrf default

       interface vlan 1

       interface vlan 4000

       interface vlan 1 ip address 192.168.1.254 255.255.255.0

       interface vlan 4000 ip address 10.10.10.1 255.255.255.0

       dcb priority-flow-control enable force

       interface ethernet 1/35 dcb priority-flow-control mode on force

       interface ethernet 1/36 dcb priority-flow-control mode on force

       interface port-channel 1 dcb priority-flow-control mode on force

       mlag-vip my-new-domain ip 10.20.2.205 /24 force

    no mlag shutdown

       mlag system-mac 00:00:5E:00:01:5D

       interface port-channel 1 ipl 1

       interface vlan 4000 ipl 1 peer-address 10.10.10.2

     

    Connectivity

    After the configuration is done, connect the Mellanox switch pair to the Cisco Nexus vPC pair as shown in the setup section above. At this time, all Mellanox interfaces should be in "down" state, since mlag-port-channel 100 is in manual shutdown.

     

    Configure interface mlag-port-channel 100 no shutdown command at both Mellanox switches.

     

    From the Cisco Nexus vPC side, you will see the log output below.

    • Interface Ethernet 1/1 and 1/2 will change from down to up.
    • Port-channel 100 will become up.
    • From the Cisco Nexus spanning tree side, you will see port-channel 100 change from BLK to LRN to FWD.

     

    Nexus-1# 2011 Sep 19 05:25:18 Nexus-1 %ETHPORT-5-SPEED: Interface Ethernet1/1, operational speed changed to 10 Gbps

    2011 Sep 19 05:25:18 Nexus-1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/1, operational duplex mode changed to Full

    2011 Sep 19 05:25:18 Nexus-1 %ETHPORT-5-SPEED: Interface port-channel100, operational speed changed to 10 Gbps

    2011 Sep 19 05:25:18 Nexus-1 %ETHPORT-5-IF_DUPLEX: Interface port-channel100, operational duplex mode changed to Full

    2011 Sep 19 05:25:19 Nexus-1 %ETHPORT-5-SPEED: Interface Ethernet1/2, operational speed changed to 10 Gbps

    2011 Sep 19 05:25:19 Nexus-1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/2, operational duplex mode changed to Full

    2011 Sep 19 05:25:19 Nexus-1 %ETHPORT-5-SPEED: Interface port-channel100, operational speed changed to 10 Gbps

    2011 Sep 19 05:25:19 Nexus-1 %ETHPORT-5-IF_DUPLEX: Interface port-channel100, operational duplex mode changed to Full

    2011 Sep 19 05:25:22 Nexus-1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel100: Ethernet1/1 is up

    2011 Sep 19 05:25:22 Nexus-1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel100: first operational port changed from none to Ethernet1/1

    2011 Sep 19 05:25:22 Nexus-1 %ETHPORT-5-IF_UP: Interface Ethernet1/1 is up in mode access

    2011 Sep 19 05:25:22 Nexus-1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel100: Ethernet1/2 is up

    2011 Sep 19 05:25:22 Nexus-1 %ETHPORT-5-IF_UP: Interface port-channel100 is up in mode access

    2011 Sep 19 05:25:22 Nexus-1 %ETHPORT-5-IF_UP: Interface Ethernet1/2 is up in mode access

     

    Get the spanning tree configuration in Nexus-1:

    Nexus-1# show spanning-tree

    VLAN0001

      Spanning tree enabled protocol rstp

      Root ID    Priority    32769

                 Address     0023.04ee.be01

                 This bridge is the root

                 Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

      Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)

                 Address     0023.04ee.be01

                 Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

    Interface        Role Sts Cost      Prio.Nbr Type

    ---------------- ---- --- --------- -------- --------------------------------

    Po1              Desg FWD 1         128.4096 (vPC peer-link) Network P2p

    Po100            Desg BLK 1         128.4195 (vPC) P2p

    Po200            Desg FWD 1         128.4295 (vPC) P2p

     

     

    Nexus-1# show spanning-tree

    VLAN0001

      Spanning tree enabled protocol rstp

      Root ID    Priority    32769

                 Address     0023.04ee.be01

                 This bridge is the root

                 Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

      Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)

                 Address     0023.04ee.be01

                 Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

    Interface        Role Sts Cost      Prio.Nbr Type

    ---------------- ---- --- --------- -------- --------------------------------

    Po1              Desg FWD 1         128.4096 (vPC peer-link) Network P2p

    Po100            Desg FWD 1         128.4195 (vPC) P2p

    Po200            Desg FWD 1         128.4295 (vPC) P2p

     

    Run the following on the Mellanox switches:

    mti-mar-sx03 [my-new-domain: master] (config) # show int mlag-port-channel summary

    MLAG Port-Channel Flags: D-Down, U-Up

                             P-Partial UP, S - suspended by MLAG

    Port Flags: D - Down, P - Up in port-channel (members)

                S - Suspend in port-channel (members), I - Individual

     

    Group

    Port-Channel      Type       Local Ports              Peer Ports

    (D/U/P/S)                    (D/P/S/I)                (D/P/S/I)

    --------------------------------------------------------------------------------

    1 Mpo100(U)       LACP       Eth1/1(P), Eth1/2(P)     Eth1/1(P), Eth1/2(P)

     

    Troubleshooting Scenarios

    Error Scenario 1: Mellanox MLAG Switches are connected to Nexus vPC before setting up the mlag-port-channel

    If no mlag-port-channel is configured, Mellanox switches running without spanning tree will cause network a loop.

    In this case, Ethernet 1/1 and 1/2 will be in FWD state on the Cisco Nexus side and any broadcast and unknown unicast will forward infinitely in the network.

     

    Error Scenario 2: MLAG pair configuration does not match (during the configuration process)

    If the configuration of mlag-port-channel on the MLAG switch pair is not matching a network loop is created. In this case mlag-port-channel 100 is up on SX04, for example, but on SX03, it is not configured yet.

    This causes any broadcast and unknown unicast to forward infinitely in the network.

     

    For example:

    on SX04

    mti-mar-sx04 [my-new-domain: standby] # show interface mlag-port-channel summary

    MLAG Port-Channel Flags: D-Down, U-Up

                             P-Partial UP, S - suspended by MLAG

    Port Flags: D - Down, P - Up in port-channel (members)

                S - Suspend in port-channel (members), I - Individual

    Group

    Port-Channel      Type       Local Ports              Peer Ports

    (D/U/P/S)                    (D/P/S/I)                (D/P/S/I)

    --------------------------------------------------------------------------------

    1 Mpo100(P)       LACP       Eth1/1(P), Eth1/2(P)     Eth1/1(D), Eth1/2(D)

     

    on SX03:

    mti-mar-sx03 [my-new-domain: master] (config) # show interface mlag-port-channel summary

    MLAG Port-Channel Flags: D-Down, U-Up

                             P-Partial UP, S - suspended by MLAG

    Port Flags: D - Down, P - Up in port-channel (members)

                S - Suspend in port-channel (members), I - Individual

     

    Group

    Port-Channel      Type       Local Ports              Peer Ports

    (D/U/P/S)                    (D/P/S/I)                (D/P/S/I)

    --------------------------------------------------------------------------------

    1 Mpo100(P)       Unknown                             Eth1/1(P), Eth1/2(P)

     

    On Nexus-1

    Nexus-1# show spanning-tree

    VLAN0001

      Spanning tree enabled protocol rstp

      Root ID    Priority    32769

                 Address     0023.04ee.be01

                 This bridge is the root

                 Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

      Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)

                 Address     0023.04ee.be01

                 Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

     

    Interface        Role Sts Cost      Prio.Nbr Type

    ---------------- ---- --- --------- -------- --------------------------------

    Po1              Desg FWD 1         128.4096 (vPC peer-link) Network P2p

    Po100            Desg FWD 1         128.4195 (vPC) P2p

    Po200            Desg FWD 1         128.4295 (vPC) P2p

    Eth1/1           Desg FWD 2         128.129  P2p

     

    Note: On the Cisco side, you will see the following output:

     

    Nexus-1# 2011 Sep 19 01:28:02 Nexus-1 %FWM-2-STM_LOOP_DETECT: Loops detected in the network for mac e41d.2d26.3bc1 among ports Po100 and Eth1/1 vlan 1 - Disabling dynamic learn notifications for 180 seconds

     

     

     

     

    Error Scenario 3: MLAG-port-channel does not come up correctly

    For some reason, mlag-port-channel may not come up correctly. So in this instance, SX03 mlag-port-channel is down while SX04 mlag-port-channel is up.

    mti-mar-sx03 [my-new-domain: master] (config) # show int mlag-port-channel 100

    Mpo100

      Admin state: Enabled

      Operational state: Down

      Description: N\A

      Mac address: e4:1d:2d:37:55:80  

      MTU: 1500 bytes(Maximum packet size 1522 bytes)

      lacp-individual mode: Disabled

      Flow-control: receive off send off

      Actual speed: 2 X 10 Gbps        

      Width reduction mode: Not supported

      Switchport mode: access

      MAC learning mode: Enabled

      Last clearing of "show interface" counters : Never              

      60 seconds ingress rate: 0 bits/sec, 0 bytes/sec, 0 packets/sec

      60 seconds egress rate: 0 bits/sec, 0 bytes/sec, 0 packets/sec

    mti-mar-sx03 [my-new-domain: master] (config) # show int mlag-port-channel summary 

    MLAG Port-Channel Flags: D-Down, U-Up

                             P-Partial UP, S - suspended by MLAG

    Port Flags: D - Down, P - Up in port-channel (members)

                S - Suspend in port-channel (members), I - Individual

     

     

    Group

    Port-Channel      Type       Local Ports              Peer Ports

    (D/U/P/S)                    (D/P/S/I)                (D/P/S/I)

    --------------------------------------------------------------------------------

    1 Mpo100(S)       LACP       Eth1/1(D), Eth1/2(D)     Eth1/1(P), Eth1/2(P)

     

    To resolve this, restart the MLAG process manually. Run:

    switch (config) # mlag shutdown

    switch (config) # mlag no shutdown

     

    Error Scenario 4: Cisco vPC port-channel in hot standby mode

    For some reason, Cisco vPC port-channel may be in hot standby mode. In this case, on the Mellanox side, mlag-port-channel is in suspend mode.

     

    For example:

    On Nexus-1

    Nexus-1# show port-channel summary 

    Flags:  D - Down        P - Up in port-channel (members)

            I - Individual  H - Hot-standby (LACP only)

            s - Suspended   r - Module-removed

            S - Switched    R - Routed

            U - Up (port-channel)

            M - Not in use. Min-links not met

    --------------------------------------------------------------------------------

    Group Port-       Type     Protocol  Member Ports

          Channel

    --------------------------------------------------------------------------------

    1     Po1(SU)     Eth      LACP      Eth2/3(P)    Eth2/4(P)   

    100   Po100(SU)   Eth      LACP     Eth1/1(H)    Eth1/2(P)  

     

    On SX03

    mti-mar-sx03 [my-new-domain: master] (config) # show int mlag-port-channel summary

    MLAG Port-Channel Flags: D-Down, U-Up

                             P-Partial UP, S - suspended by MLAG

    Port Flags: D - Down, P - Up in port-channel (members)

                S - Suspend in port-channel (members), I - Individual

     

    Group

    Port-Channel      Type       Local Ports              Peer Ports

    (D/U/P/S)                    (D/P/S/I)                (D/P/S/I)

    --------------------------------------------------------------------------------

    1 Mpo100(D)       LACP      Eth1/1(S), Eth1/2(S)     Eth1/1(P), Eth1/2(P)

     

    To resolve this, restart the MLAG process manually. Run:

    switch (config) # mlag shutdown

    switch (config) # mlag no shutdown