HowTo Get Started with Mellanox switches

Version 8

    This post is basic, and meant for beginners who want to get started with Mellanox Switches (Ethernet or InfiniBand) and perform basic configuration tasks.

    The following procedure and examples are given from Mellanox Ethernet switch SX1710, but it is similar to other switches.

     

    References

     

    SX1710 -- Front View

    sx1710.PNG

    SX1710 -- Back View

    back.jpg

     

    Configuration

    1. Connect to the console port of the switch to a COM port of the server to configure the basic parameters.

    Note: Check the baud rate and other RS232 parameters of the console port of the Switch (refer to the appropriate Hardware User Manual).

     

    console.jpg

     

    For example:

    The SX1710  baud rate is: 115200

     

    First time username: admin

    First time password: admin

     

    2. Run the first wizard to set the hostname and DHCP. Here is an example of output you might see:

    ...

     

    Mellanox MLNX-OS Switch Management

     

    Password:

     

    Mellanox Switch

     

    Mellanox configuration wizard

     

    Do you want to use the wizard for initial configuration? y

     

    Step 1: Hostname? [switch-5fa5a0] my-switch-hostname

    Step 2: Use DHCP on mgmt0 interface? [yes]

    Step 3: Enable IPv6? [yes] no

    Step 4: Admin password (Enter to leave unchanged)?

     

    You have entered the following information:

     

       1. Hostname: my-switch-hostname

       2. Use DHCP on mgmt0 interface: yes

       3. Enable IPv6: no

       4. Admin password (Enter to leave unchanged): (unchanged)

     

    To change an answer, enter the step number to return to.

    Otherwise hit <enter> to save changes and exit.

     

    Choice:

     

    Configuration changes saved.

     

    3. Map the MGT1: MAC address of the management port (MGT1) located on the label to the DHCP server (assign static IP address for that MAC).

     

    Note: Notice that there are 3 MAC addresses on the label. Make sure you map the correct one labeled with MGT1, see below.

    DHCP Server Label

    label3-1.jpg

     

    4. Connect the management port (MGT1) to the management network.

    mgmt1.jpg

     

    At this point the switch should retrieve the IP address from the DHCP server. Now you can connect using ssh by including the given IP or switch hostname.

     

    5. Connect remotely using ssh.

    - Check that both the admin port and link are up.

    - Verify the IP Address.

    - Verify the MAC address.

     

    # ssh admin@10.20.2.54

     

    Mellanox MLNX-OS Switch Management

     

    Password:

    Last login: Sun Apr 26 10:52:11 2015 from 10.208.0.121

     

    Mellanox Switch

     

    my-switch-hostname > enable

    my-switch-hostname # configure terminal

    my-switch-hostname (config) #

    my-switch-hostname (config) # show interfaces mgmt0

    Interface mgmt0 status:

       Comment:

       Admin up:           yes

       Link up:            yes

       DHCP running:       yes

       IP address:         10.20.2.54

       Netmask:            255.255.0.0

       IPv6 enabled:       no

       Speed:              N/A

       Duplex:             N/A

       Interface type:     ethernet

       Interface source:   bridge

       MTU:                1500

       HW address:         E4:1D:2D:5F:A7:C0

     

     

       RX bytes:           744857403           TX bytes:       41239629

       RX packets:         2875964             TX packets:     117768

       RX mcast packets:   0                   TX discards:    0

       RX discards:        0                   TX errors:      0

       RX errors:          0                   TX overruns:    0

       RX overruns:        0                   TX carrier:     0

       RX frame:           0                   TX collisions:  0

                                               TX queue len:   0                          

     

     

    Basic CLI Commands

    Here is a basic list of commands to start:

     

    1. Enter the "enable" and "configure terminal" commands each time you get into the switch so that you can enter the configuration mode.

    # ssh admin@10.20.2.54

     

    Mellanox MLNX-OS Switch Management

     

    Password:

    Last login: Sun Apr 26 10:52:11 2015 from 10.208.0.121

     

    Mellanox Switch

     

    my-switch-hostname  > enable

    my-switch-hostname  # configure terminal

    my-switch-hostname (config) #

     

    2. If you are running a control lab environment, you might want to remove the admin password and auto-logout settings as follows:

    my-switch-hostname (config) # username admin nopassword

    my-switch-hostname (config) # no cli default auto-logout

     

    3. Use the "configuration write" command to save the configuration to the startup-config file. Unless you run this command, upon restart the running configuration settings will be deleted.

    my-switch-hostname (config) # configuration write

     

    4. Use the "show running-config" command to display the current configuration on the switch.

    my-switch-hostname (config) # show running-config
    ....

     

    5. Instead of using the Linux "grep" command, you can use "|"  and "include" or "exclude" to include or exclude text from the output.

    For example:

    my-switch-hostname (config) # show running-config | include ethernet
    ....

     

    6. Use the "show interface status - Ethernet" command to display status information.

    Note: This command is applicable only for Ethernet switches or Gateway (for the Ethernet ports).

    my-switch-hostname (config) # show interfaces ethernet status

     

    Port                   Operational state           Speed                  Negotiation

    ----                   -----------------           -----                  -----------

    Eth1/1                 Up                          40 Gbps             No-Negotiation

    Eth1/2                 Down                        40 Gbps             No-Negotiation

    Eth1/3                 Down                        40 Gbps             No-Negotiation

     

     

    7. Use the "show interface status - InfiniBand" command to display status information.

    Note: This command is applicable only for InfiniBand switches or Gateway (for the InfiniBand ports).

    my-switch-hostname (config) # show interfaces ib status

     

    Interface      Description                                Speed                   Current line rate   Logical port state   Physical port state

    ---------      -----------                                ---------               -----------------   ------------------   -------------------

    Ib 1/1                                                    2.5 Gbps rate only      10.0 Gbps           Down                 Polling

    Ib 1/2                                                    14.0 Gbps rate          56.0 Gbps           Active               LinkUp

    Ib 1/3                                                    2.5 Gbps rate only      10.0 Gbps           Down                 Polling

    Ib 1/4                                                    14.0 Gbps rate          56.0 Gbps           Active               LinkUp

    Ib 1/5                                                    2.5 Gbps rate only      10.0 Gbps           Down                 Polling

     

    Refer to the MLNX-OS User Manual for all switch features and commands.

     

    Troubleshooting

    1. Make sure to use MGT1 MAC address on the DHCP server.

    2. Note that MGT1 port is mapped to mgmt0 port on the MLNX-OS switch software.

    3. Note that in systems with two power supplies (such as the SX1710), you are expected to connect both power supplies to the power system, otherwise the power and system LEDs will be red (bad status).  You can clear the red LED by simply unplugging the second (and unused) power tray.

    rack behind1-1.jpg

     

    4. To re-run the initial wizard, enter the run command:

     

    switch (config) # configuration jump-start

     

    Mellanox configuration wizard

     

    Step 1: Hostname? [switch]

    Step 2: Use DHCP on mgmt0 interface? [yes]

    Step 3: Enable IPv6? [no]

    Step 4: Admin password (Enter to leave unchanged)?

     

    You have entered the following information:

       1. Hostname: switch

       2. Use DHCP on mgmt0 interface: yes

       3. Enable IPv6: no

       4. Admin password (Enter to leave unchanged): (unchanged)

     

    To change an answer, enter the step number to return to.

    Otherwise hit <enter> to save changes and exit.

     

    Choice:

    Configuration changes saved.

    switch  (config) #