Running SONiC on Mellanox Spectrum Switches

Version 9

    This is a short guide on how to deploy SONiC on a Mellanox switch.

     

    If you are not familiar with SONiC check out this blog post to understand its high level architecture.

     

    SONiC can be deployed on any switch in the Mellanox Ethernet portfolio.

     

     

     

    SN2700 – 32x40/100G or 64x10/25G

     

    SN2410 – 48x10/25G + 8x40G/100G

     

    SN2100 – 16x40/100G or 64x10/25G

     

     

    Precondition for SONiC installation

    Start by checking the BIOS and ONIE version:

    1.   Connect to switch via serial console.

      Check that your SMIBIOS and ONIE Version are updated.

    SMIBIOS:

                   1. Check the Product name and SKU Number

     

    #sudo su
    # apt-get update && apt-get install -y dmidecode

    # dmidecode -t1
    System Information
    Manufacturer: Mellanox Technologies Ltd.

    Product Name: Mellanox switch
    Version: Defined in system VPD
    Serial Number: Defined in system VPD
    UUID: 03000200-0400-0500-0006-000700080009 Wake-up

    Type: Power Switch
    SKU Number: Defined in system VPD
    Family: Mellanox SwitchX family

     

    Note: If Product Name and SKU Number includes MSN2XXX than you can skip the SMIBIOS update if not follow the next steps.

            

                   2. Locate amifldrv_mod.o.

     

    ONIE:/ # find / -name amifldrv_mod.o /usr/bin/amifldrv_mod.o

            

                   3. Create a soft link

     

    ONIE:/ # ln –s amifldrv_mod.o amifldrv_mod.ko

            

                   4. Update Product name and SKU Number (Note: make sure amidelnx_64 is executable), you can find the SKU Number on the switch label

     

    ONIE:/ # ./amidelnx_64 /SP MSN2700-CS2FO /SK MSN2700-CS2FO

     

                   5. Verify Product name and SKU Number

     

    ONIE:/ # dmidecode -t1

    # dmidecode 2.12 SMBIOS 2.7 present.

    Handle 0x0001, DMI type 1, 27 bytes System Information

    Manufacturer: Mellanox Technologies Ltd.

    Product Name: MSN2700-CS2FO

    Version: Defined in system VPD
    Serial Number: Defined in system VPD
    UUID: 03000200-0400-0500-0006-000700080009 Wake-up

    Type: Power Switch
    SKU Number: MSN2700-CS2FO
    Family: Mellanox SwitchX family

     

     

    ONIE Version:

    There are 2 versions (different baud rates) for ONIE to support Sonic single image. On both ONIE will set mac address based on eeprom of eth0 and not based on MAC in onie-eeprom.

    Versions: 2016.11-5.1.0012-9600 and 2016.11-5.1.0012-115200

    If your onie version is older than the ones specific above, you should upgrade ONIE.

     

    ONIE Update:

     

                   1. Enter ONIE Rescue mode.

     

                   2. Update machine_rev parameter to 0 in /etc/machine.conf:

    ONIE:/ # vi /etc/machine.conf

     

    onie_version=HEAD-201502020016

    onie_vendor_id=33049

    onie_platform=x86_64-mlnx_x86-r0

    onie_machine=mlnx_x86

    onie_machine_rev=0

    onie_arch=x86_64

    onie_config_version=1

    onie_build_date="2015-02-02T00:16+0200"

    onie_partition_type=gpt

    onie_kernel_version=0

            

                   3.Copy the onie-updater file to the switch (or use the URL/TFTP/etc)

            

                   4.Update the ONIE image:

    ONIE:/ # onie-self-update onie-updater-x86_64-mlnx_x86-r0

     

    Now we are ready for the SONiC NOS install.

     

    Install SONiC

     

         1. (Optional) Some switches may come with a NOS which will require you to uninstall the existing NOS first before you install SONiC. To do so, simply boot into ONIE and select Uninstall OS:

     

                         GNU GRUB  version 2.02~beta2+e4a1fe391

    +----------------------------------------------------------------------------+

    |*ONIE: Install OS |

    | ONIE: Rescue                                                               |

    | ONIE: Uninstall OS  <----- Select this one                                 |

    | ONIE: Update ONIE |

    | ONIE: Embed ONIE |

    +----------------------------------------------------------------------------+

     

        2.Reboot the switch into ONIE and select Install OS:

     

            GNU GRUB  version 2.02~beta2+e4a1fe391

    +----------------------------------------------------------------------------+

    |*ONIE: Install OS    <----- Select this one                                 |

    | ONIE: Rescue                                                               |

    | ONIE: Uninstall OS                                                         |

    | ONIE: Update ONIE                                                          |

    | ONIE: Embed ONIE                                                           |

    +----------------------------------------------------------------------------+

     

        3.if your Switch is connected to DHCP you can skip the next step of configuring a static IP address.

       ONIE:/ # ifconfig eth0 192.168.0.2 netmask 255.255.255.0

    ONIE:/ # ip route add default via 192.168.0.1

     

    The OS can be pulled to the switch in multiple ways:

     

    a.   Copy the image to the switch with SCP.

    b.   Upload the image to your local http server and pull from there.

    c.   Pull the OS from the Web directly if the switch has access to the internet.

     

    The latest image file for Mellanox switches can be found here:

     

    https://sonic-jenkins.westus2.cloudapp.azure.com/job/mellanox/job/buildimage-mlnx-all/lastSuccessfulBuild/artifact/target/sonic-mellanox.bin

     

    Note: This is the last successful build and doesn’t necessarily mean it successfully passed all the tests.

    (only the build process was successful)

     

    Install the OS

     

    ONIE:/ # onie-nos-install http://192.168.2.10/sonic-mellanox.bin

    *When NOS installation finishes, the box will reboot into SONiC by default.

     

    Note: By default, the SONiC console baud rate is 9600. You may need to change the baud rate in case you cannot see anything from the console after reboot.

    You must have BIOS and ONIE that supports it

     

    The Default Username is admin and the password is YourPaSsWoRd to login for the first time.

     

    Lets explore SONiC

     

    Check that you are able to see all of the switch interfaces

    admin@r-stav02:~$ show interfaces description

    Command: intfutil description

    Interface    Oper    Admin Alias    Description

    ----------- ------  -------  ---------- -------------

    Ethernet0      up       up Ethernet0            N/A

    Ethernet4      up       up Ethernet4            N/A

    Ethernet8    down     down Ethernet8            N/A

    Ethernet12 down     down  Ethernet12            N/A

    Ethernet16 down     down  Ethernet16            N/A

    Ethernet20 down     down  Ethernet20            N/A

    Ethernet24 up       up  Ethernet24            N/A

    Ethernet28 up       up  Ethernet28            N/A

    Ethernet32 down     down  Ethernet32            N/A

    Ethernet36 down     down  Ethernet36            N/A

    Ethernet40 down     down  Ethernet40            N/A

    Ethernet44 down     down  Ethernet44            N/A

    Ethernet48 down     down  Ethernet48            N/A

    Ethernet52 down     down  Ethernet52            N/A

    Ethernet56 down     down  Ethernet56            N/A

    Ethernet60 down     down  Ethernet60            N/A

     

     

    The SONiC Config file:

     

    The location of the SONiC config file is on the switch is:


    /etc/sonic/config_db.json

     

    When the switch boots, it loads the data from the file into the Redis database. The switch daemons that are running in Docker monitor the database and update their operating state.

     

    You can find examples of how to change configurations via the file here:

    https://github.com/Azure/SONiC/wiki/Configuration

     

    Let’s edit the file and run BGP.

     

     

     

     

    In my setup I have 4 BGP neighbours connected to my switch on the following ports and IPs

    Open the config file:

     

    sudo vi /etc/sonic/config_db.json

     

    Configure the Interfaces

     

        "INTERFACE": {

            "Ethernet4|40.0.0.1/30": {},

            "Ethernet24|30.0.0.14/30": {},

            "Ethernet28|30.0.0.10/30": {},

            "Ethernet0|40.0.0.5/30": {}

        },

     

     

    We will set a loopback, the BGP router ID is taken from the loopback address in SONiC by default.

     

           "LOOPBACK_INTERFACE": {

             "Loopback0|50.30.30.30 /32": {}

              },

     

    Configure the BGP neighbours

     

    "BGP_NEIGHBOR": {

             "40.0.0.6": {

                "rrclient": "0",

                "name": "OX2",

                "local_addr": "40.0.0.5",

                "nhopself": "0",

                "admin_status": "up",

                "holdtime": "180",

                "asn": "65002",

                "keepalive": "60"

            },

            "30.0.0.13": {

                "rrclient": "0",

                "name": "CORE1",

                "local_addr": "30.0.0.12",

                "nhopself": "0",

                "admin_status": "up",

                "holdtime": "180",

                "asn": "2701",

                "keepalive": "60"

            },

            "40.0.0.2": {

                "rrclient": "0",

                "name": "OX1",

                "local_addr": "40.0.0.1",

                "nhopself": "0",

                "admin_status": "up",

                "holdtime": "180",

                "asn": "65002",

                "keepalive": "60"

            },

            "30.0.0.9": {

                "rrclient": "0",

                "name": "CORE2",

                "local_addr": "30.0.0.10",

                "nhopself": "0",

                "admin_status": "up",

                "holdtime": "180",

                "asn": "2701",

                "keepalive": "60"

            }

        },

     

     

    The local AS number is part of the DEVCIEMETADATA

     

    "DEVICE_METADATA": {

    "localhost": {

                "hwsku": "ACS-MSN2100",

                "hostname": "sonic",

                "mac": "24:8a:07:3d:51:00",

                "bgp_asn": "65100",

                "deployment_id": "None",

                "type": "LeafRouter"

            }

        },

     

     

     

     

    Load the new config file into the RedisDB

     

    admin@r-stav02:~$ sudo config reload

     

    Check that BGP is up:

     

    root@r-stav02:/home/admin#show ip bgp summary

    Command: sudo vtysh -c "show ip bgp summary"

    BGP router identifier 50.30.30.30, local AS number 65100

    RIB entries 67, using 7504 bytes of memory

    Peers 4, using 18 KiB of memory

     

    Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd

    30.0.0.9        4  2701      18      15        0    0    0 00:10:50       33

    30.0.0.13       4  2701      18      18        0    0    0 00:10:50       33

    40.0.0.2        4 65002      19      18        0    0    0 00:10:48        0

    40.0.0.6        4 65002      18      18        0    0    0 00:10:47        0

    Total number of neighbors 4

     

     

     

    root@r-stav02:/home/admin# show ip route

    Command: sudo vtysh -c "show ip route"

    Codes: K - kernel route, C - connected, S - static, R - RIP,

           O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,

           > - selected route, * - FIB route

     

    K>* 0.0.0.0/0 via 10.209.20.1, eth0

    C>* 10.209.20.0/22 is directly connected, eth0

    B>* 20.0.20.0/24 [20/0] via 30.0.0.9, Ethernet28, src 50.30.30.30, 00:10:54

      *                     via 30.0.0.13, Ethernet24, src 50.30.30.30, 00:10:54

    B>* 20.0.21.0/24 [20/0] via 30.0.0.9, Ethernet28, src 50.30.30.30, 00:10:54

      *                     via 30.0.0.13, Ethernet24, src 50.30.30.30, 00:10:54

    B>* 20.0.22.0/24 [20/0] via 30.0.0.9, Ethernet28, src 50.30.30.30, 00:10:54

      *                     via 30.0.0.13, Ethernet24, src 50.30.30.30, 00:10:54

    B>* 20.0.23.0/24 [20/0] via 30.0.0.9, Ethernet28, src 50.30.30.30, 00:10:54

      *                     via 30.0.0.13, Ethernet24, src 50.30.30.30, 00:10:54

     

     

     

    It is possible to see all of the resources used on the switch

     

    admin@r-stav02:~$ crm show resources all

    Resource Name           Used Count    Available Count

    --------------------  ------------  -----------------

    ipv4_route                      43             122825

    ipv6_route                       1             122825

    ipv4_nexthop                     4              57342

    ipv6_nexthop                     0              57342

    ipv4_neighbor                    4             122825

    ipv6_neighbor                    0              24576

    nexthop_group_member             2              57342

    nexthop_group                    1              57342

    fdb_entry                        0             122825

     

    Stage    Bind Point    Resource Name      Used Count    Available Count

    -------  ------------  ---------------  ------------  -----------------

    INGRESS  PORT          acl_group                   0                399

    INGRESS  PORT          acl_table                   0                393

    INGRESS  LAG           acl_group                   0                399

    INGRESS  LAG           acl_table                   0                393

    INGRESS  VLAN          acl_group                   0                399

    INGRESS  VLAN          acl_table                   0                393

    INGRESS  RIF           acl_group                   0                399

    INGRESS  RIF           acl_table                   0                393

    INGRESS  SWITCH        acl_group                   0                399

    INGRESS  SWITCH        acl_table                   0                393

    EGRESS   PORT          acl_group                   0                399

    EGRESS   PORT          acl_table                   1                393

    EGRESS   LAG           acl_group                   0                399

    EGRESS   LAG           acl_table                   0                393

    EGRESS   VLAN          acl_group                   0                  0

    EGRESS   VLAN          acl_table                   0                  0

    EGRESS   RIF           acl_group                   0                399

    EGRESS   RIF           acl_table                   0                393

    EGRESS   SWITCH        acl_group                   0                399

     

    EGRESS   SWITCH        acl_table                   0                393

     

     

     

    • CRM allows to configure polling interval (in seconds):
    • # crm config polling interval 300
    • CRM allows to configure threshold type:
    • # crm config thresholds ipv4 route type percentage
    • CRM allows to configure threshold values:
    • # crm config thresholds ipv4 route low 70
    • # crm config thresholds ipv4 route high 85

    Syslog message when high threshold exceeded:

    • IPV4_ROUTE THRESHOLD_EXCEEDED for TH_USED 0% Used count 24 free count 57218
    • Syslog message when low threshold exceeded:
    • IPV4_ROUTE THRESHOLD_CLEAR for TH_USED 0% Used count 24 free count 57218

     

    You can find all the show commands supported in the SONIC CLI:

     

    admin@r-stav02:~$ show

    Commands:

      aaa                   Show AAA configuration in ConfigDb

      acl                   Show ACL related information

      arp                   Show IP ARP table

      clock                 Show date and time

      ecn                   Show ECN configuration

      environment           Show environmentals (voltages, fans, temps)

      interfaces            Show details of the network interfaces

      ip                    Show IP (IPv4) commands

      ipv6                  Show IPv6 commands

         lldp                  LLDP (Link Layer Discovery Protocol)...

      logging               Show system log

      mac                   Show MAC (FDB) entries

      ntp                   Show NTP information

      platform              Show platform-specific hardware info

      processes             Display process information

      runningconfiguration  Show current running configuration...

      services              Show all daemon services

      session               Show existing everflow sessions

      startupconfiguration  Show startup configuration information

      system-memory         Show memory information

      tacacs                Show TACACS+ configuration

      techsupport           Gather information for troubleshooting

      uptime                Show system uptime

      users                 Show users

      version               Show version information

      vlan                  Show VLAN information

     

     

    config commands:

     

    admin@r-stav02:~$ config

    Commands:

      aaa               AAA command line

      acl               ACL-related configuration tasks

      bgp               BGP-related configuration tasks

      ecn               ECN-related configuration tasks

      interface         Interface-related configuration tasks

      load              Import a previous saved config DB dump file.

      load_mgmt_config  Reconfigure hostname and mgmt interface based...

      load_minigraph    Reconfigure based on minigraph.

      qos

      reload            Clear current configuration and import a...

      save              Export current config DB to a file on disk.

      tacacs            TACACS+ server configuration

      vlan              VLAN-related configuration tasks

     

     

     

    use –-help on each config command in order to understand the exact syntax and usage.

     

    admin@r-stav02:~$ sudo config ecn --help

    Usage: config ecn [OPTIONS]

      ECN-related configuration tasks

    Options:

      -profile <profile_name>       Profile name  [required]

      -rmax <red threshold max>     Set red max threshold

      -rmin <red threshold min>     Set red min threshold

      -ymax <yellow threshold max>  Set yellow max threshold

      -ymin <yellow threshold min>  Set yellow min threshold

      -gmax <green threshold max>   Set green max threshold

      -gmin <green threshold min>   Set green min threshold

     

     

    when using config commands, you will need to execute “config save” in order to apply configuration into the ConfigDB and then “config reload” to update the RedisDB.

     

     

    SONiC is based on Containers running its services

     

     

    You can see all of the Containers running by executing the command bellow

     

    admin@r-stav02:~$ docker ps --format 'table {{.Names}}\t{{.Command}}\t{{.Image}}'

    NAMES               COMMAND                  IMAGE

    dhcp_relay          "/usr/bin/docker_init"   docker-dhcp-relay:latest

    snmp                "/usr/bin/supervisord"   docker-snmp-sv2:latest

    syncd               "/usr/bin/supervisord"   docker-syncd-mlnx:latest

    swss                "/usr/bin/supervisord"   docker-orchagent-mlnx:latest

    teamd               "/usr/bin/supervisord"   docker-teamd:latest

    lldp                "/usr/bin/supervisord"   docker-lldp-sv2:latest

    pmon                "/usr/bin/supervisord"   docker-platform-monitor:latest

    bgp                 "/usr/bin/supervisord"   docker-fpm-quagga:latest

    database            "/usr/bin/supervisord"   docker-database:latest

     

     

    You can access a container by executing the following command

     

    admin@r-stav02:~$ docker exec -it bgp bash

    root@sonic:/#  

     

     

     

    In SONiC, most of the critical network and system data is stored in redisDB.

    DB name

    DB No.

    Description

    APPL_DB

    0

    Application running data

    ASIC_DB

    1

    ASIC configuration and state data

    COUNTERS_DB

    2

    Counter data for port, lag, queue

    LOGLEVEL_DB

    3

    Log level control for SONiC modules

    CONFIG_DB

    4

    Source of truth for SONiC configuration

    FLEX_COUNTER_DB

    5

    For PFC watch dog counters control and other plugin extensions

    STATE_DB

    6

    Configuration state for object in CONFIG_DB

     

    Checking the RedisDB state can be done as follows:

     

    root@r-stav02:/home/admin#redis-cli

    127.0.0.1:6379> KEYS INTF*

    1) "INTF_TABLE:Ethernet0:40.0.0.5/30"

    2) "INTF_TABLE:Ethernet4:40.0.0.1/30"

    3) "INTF_TABLE:eth0:10.209.22.49/22"

    4) "INTF_TABLE:docker0:240.127.1.1/24"

    5) "INTF_TABLE:lo:50.30.30.30"

    6) "INTF_TABLE:Ethernet28:30.0.0.10/30"

    7) "INTF_TABLE:Ethernet24:30.0.0.14/30"

     

    more examples can be seen in the following link:

    https://github.com/Azure/sonic-swss/blob/4c56d23b9ff4940bdf576cf7c9e5aa77adcbbdcc/doc/swss-schema.md

     

     

    Splitting a port on SONiC

     

    The port split feature allows to extend number of available switch ports by splitting one port up to 4 sub-ports (1 per lane).

     

    Example of supported split options:

    1.   1x100G -> 2x50G

    2.   1x100G -> 4x25G

    3.   1x100G -> 4x10G

     

    Note: Please review the HW user manual in order of understanding which ports can be split, as an example on the SN2700 only the odd ports can be split, and the adjacent even port should be removed from the port_config.ini

     

      e.g. if we want to split port Ethernet0 we will need to remove port Ethernet1 from the port_config.ini

    http://www.mellanox.com/related-docs/user_manuals/1U_HW_UM_SN2000_Switch_Family.pdf

     

    Port split can be configured only in port_config.ini configuration file.

    • SWSS create ports according to the described ports in this file.
    • Ports that are not specified in the file will not be created.

     

    Example of port_config.ini:

     


    vim /usr/share/sonic/device/x86_64-mlnx_msn2100-r0/ACS-MSN2100/port_config.ini
    # name         lanes         speed
    Ethernet0      0,1,2,3
    ...

     

    Note: port_config.ini should not contain blank lines because according to current behavior SONiC will throw an error an abort

     

    Example on how to split port Ethernet0 into 2 ports :

    • Add new port to the list in port_config.ini (e.g. Ethernet2)
    • Reallocate hardware lanes: Ethernet0 – 0,1; Ethernet2 – 2,3
    • Set default speed in example to 50000 (50G)
    • Apply configuration

     

    Example of “1x100 -> 2x50” split configuration:

    vim /usr/share/sonic/device/x86_64-mlnx_msn2100-r0/ACS-MSN2100/port_config.ini
    # name         lanes         speed
    Ethernet0      0,1           50000
    Ethernet2      2,3           50000
    ...

     

    Note: Speed is mandatory for split ports. If not available, ports will not be created

    Note: The max speed must not exceed 100g for all split ports. On such case ports will not be created

     

    How to apply changes in port_config.ini?


    There are 2 options,

    1.   Load Minigraph by executing‘config load_minigraph’, which will create new config_db.json based on the Minigraph

    2.   Align config_db json with the changes done in port_config.ini  and reload the configuration.

     

    Once split is done defaults port’s attributes are used (for example: MTU, speed, admin mode).
    In case non-default values are required need to configure new (sub)ports.
    It can be done like for any other regular not split ports:

    • Through the mingraph.xml
    • In config_db.json configuration file
    • Using Linux CLI commands

     

    SONiC Telemetry

    Being able to get the underlying characteristics of the network devices - either operational state or configuration, efficiently and quickly,

    will greatly facilitate the analysis of network status and improve network stability.

    Besides the traditional data collecting methods like SNMP, syslog and CLI, gRPC is the modern communication protocol supported by SONiC for telemetry streaming.

     

    Read more on how to use gRPC with SONiC:

    https://github.com/Azure/sonic-telemetry/blob/master/doc/grpc_telemetry.md

     

     

    Troubleshooting

     

    All SONiC logs are available in /var/log/syslog

    • To change log level use the swssloglevel utility
    • Set orchagent severity level to NOTICE:
      # swssloglevel -l NOTICE -c orchagent
    • Set SAI_API_SWITCH severity to ERROR:
      # swssloglevel -l SAI_LOG_LEVEL_ERROR -s -c SWITCH
    • Set all SAI_API_* severity to DEBUG:
      # swssloglevel -l SAI_LOG_LEVEL_DEBUG -s -a

     

    Q: What should be done in case the command ‘show interfaces status’ does not show any interfaces.

    A: Ports are created by orchagent process, which is part of SWSS SONiC service.

     

    1.   Check whether SWSS service is running: # systemctl status swss

    2.   Check whether swss docker container is running: # docker ps

    3.   Check whether orchagent is running in swss docker container: # docker exec -it swss supervisorctl status

    4.   Check /var/log/syslog for errors

    5.   Check port_config.ini file whether it contains valid ports description
    In example for SN2700: /usr/share/sonic/device/x86_64-mlnx_msn2700-r0/ACS-MSN2700/port_config.ini

     

    Q:What should be done in case some of the docker containers are not running?

    A: The SONiC docker containers are started by SONiC services, so to troubleshoot such a problem you need to:

    • Check whether all services are running: # systemctl status
    • Check /var/log/syslog for errors
    • Restart SONiC service which failed, for example swss: # systemctl restart swss

     

    Q: How to investigate packet drops?

    A: Packet drops can be investigated by viewing counters using the “show interfaces counters” command

    • RX_ERR/TX_ERR
    • RX_DRP
    • TX_DRP
    • RX_OVR/TX_OVR

     

     

     

    SONiC Upgrade:

     

    If you have an old SONiC version, you can easily upgrade it via the SONiC CLI:

     

    sonic_installer install https://sonic-jenkins.westus2.cloudapp.azure.com/job/mellanox/job/buildimage-mlnx-all/lastSuccessfulBuild/artifact/target/sonic-mellanox.bin

     

    once install is finished you should reboot the switch.

     

    Join the SONiC community forum !