Running SONiC on Mellanox Spectrum Switches

Version 9

    This is a short guide on how to deploy SONiC on a Mellanox switch.


    If you are not familiar with SONiC check out this blog post to understand its high level architecture.


    SONiC can be deployed on any switch in the Mellanox Ethernet portfolio.




    SN2700 – 32x40/100G or 64x10/25G


    SN2410 – 48x10/25G + 8x40G/100G


    SN2100 – 16x40/100G or 64x10/25G



    Precondition for SONiC installation

    Start by checking the BIOS and ONIE version:

    1.   Connect to switch via serial console.

      Check that your SMIBIOS and ONIE Version are updated.


                   1. Check the Product name and SKU Number


    #sudo su
    # apt-get update && apt-get install -y dmidecode

    # dmidecode -t1
    System Information
    Manufacturer: Mellanox Technologies Ltd.

    Product Name: Mellanox switch
    Version: Defined in system VPD
    Serial Number: Defined in system VPD
    UUID: 03000200-0400-0500-0006-000700080009 Wake-up

    Type: Power Switch
    SKU Number: Defined in system VPD
    Family: Mellanox SwitchX family


    Note: If Product Name and SKU Number includes MSN2XXX than you can skip the SMIBIOS update if not follow the next steps.


                   2. Locate amifldrv_mod.o.


    ONIE:/ # find / -name amifldrv_mod.o /usr/bin/amifldrv_mod.o


                   3. Create a soft link


    ONIE:/ # ln –s amifldrv_mod.o amifldrv_mod.ko


                   4. Update Product name and SKU Number (Note: make sure amidelnx_64 is executable), you can find the SKU Number on the switch label


    ONIE:/ # ./amidelnx_64 /SP MSN2700-CS2FO /SK MSN2700-CS2FO


                   5. Verify Product name and SKU Number


    ONIE:/ # dmidecode -t1

    # dmidecode 2.12 SMBIOS 2.7 present.

    Handle 0x0001, DMI type 1, 27 bytes System Information

    Manufacturer: Mellanox Technologies Ltd.

    Product Name: MSN2700-CS2FO

    Version: Defined in system VPD
    Serial Number: Defined in system VPD
    UUID: 03000200-0400-0500-0006-000700080009 Wake-up

    Type: Power Switch
    SKU Number: MSN2700-CS2FO
    Family: Mellanox SwitchX family



    ONIE Version:

    There are 2 versions (different baud rates) for ONIE to support Sonic single image. On both ONIE will set mac address based on eeprom of eth0 and not based on MAC in onie-eeprom.

    Versions: 2016.11-5.1.0012-9600 and 2016.11-5.1.0012-115200

    If your onie version is older than the ones specific above, you should upgrade ONIE.


    ONIE Update:


                   1. Enter ONIE Rescue mode.


                   2. Update machine_rev parameter to 0 in /etc/machine.conf:

    ONIE:/ # vi /etc/machine.conf













                   3.Copy the onie-updater file to the switch (or use the URL/TFTP/etc)


                   4.Update the ONIE image:

    ONIE:/ # onie-self-update onie-updater-x86_64-mlnx_x86-r0


    Now we are ready for the SONiC NOS install.


    Install SONiC


         1. (Optional) Some switches may come with a NOS which will require you to uninstall the existing NOS first before you install SONiC. To do so, simply boot into ONIE and select Uninstall OS:


                         GNU GRUB  version 2.02~beta2+e4a1fe391


    |*ONIE: Install OS |

    | ONIE: Rescue                                                               |

    | ONIE: Uninstall OS  <----- Select this one                                 |

    | ONIE: Update ONIE |

    | ONIE: Embed ONIE |



        2.Reboot the switch into ONIE and select Install OS:


            GNU GRUB  version 2.02~beta2+e4a1fe391


    |*ONIE: Install OS    <----- Select this one                                 |

    | ONIE: Rescue                                                               |

    | ONIE: Uninstall OS                                                         |

    | ONIE: Update ONIE                                                          |

    | ONIE: Embed ONIE                                                           |



        3.if your Switch is connected to DHCP you can skip the next step of configuring a static IP address.

       ONIE:/ # ifconfig eth0 netmask

    ONIE:/ # ip route add default via


    The OS can be pulled to the switch in multiple ways:


    a.   Copy the image to the switch with SCP.

    b.   Upload the image to your local http server and pull from there.

    c.   Pull the OS from the Web directly if the switch has access to the internet.


    The latest image file for Mellanox switches can be found here:



    Note: This is the last successful build and doesn’t necessarily mean it successfully passed all the tests.

    (only the build process was successful)


    Install the OS


    ONIE:/ # onie-nos-install

    *When NOS installation finishes, the box will reboot into SONiC by default.


    Note: By default, the SONiC console baud rate is 9600. You may need to change the baud rate in case you cannot see anything from the console after reboot.

    You must have BIOS and ONIE that supports it


    The Default Username is admin and the password is YourPaSsWoRd to login for the first time.


    Lets explore SONiC


    Check that you are able to see all of the switch interfaces

    admin@r-stav02:~$ show interfaces description

    Command: intfutil description

    Interface    Oper    Admin Alias    Description

    ----------- ------  -------  ---------- -------------

    Ethernet0      up       up Ethernet0            N/A

    Ethernet4      up       up Ethernet4            N/A

    Ethernet8    down     down Ethernet8            N/A

    Ethernet12 down     down  Ethernet12            N/A

    Ethernet16 down     down  Ethernet16            N/A

    Ethernet20 down     down  Ethernet20            N/A

    Ethernet24 up       up  Ethernet24            N/A

    Ethernet28 up       up  Ethernet28            N/A

    Ethernet32 down     down  Ethernet32            N/A

    Ethernet36 down     down  Ethernet36            N/A

    Ethernet40 down     down  Ethernet40            N/A

    Ethernet44 down     down  Ethernet44            N/A

    Ethernet48 down     down  Ethernet48            N/A

    Ethernet52 down     down  Ethernet52            N/A

    Ethernet56 down     down  Ethernet56            N/A

    Ethernet60 down     down  Ethernet60            N/A



    The SONiC Config file:


    The location of the SONiC config file is on the switch is:



    When the switch boots, it loads the data from the file into the Redis database. The switch daemons that are running in Docker monitor the database and update their operating state.


    You can find examples of how to change configurations via the file here:


    Let’s edit the file and run BGP.





    In my setup I have 4 BGP neighbours connected to my switch on the following ports and IPs

    Open the config file:


    sudo vi /etc/sonic/config_db.json


    Configure the Interfaces


        "INTERFACE": {

            "Ethernet4|": {},

            "Ethernet24|": {},

            "Ethernet28|": {},

            "Ethernet0|": {}




    We will set a loopback, the BGP router ID is taken from the loopback address in SONiC by default.


           "LOOPBACK_INTERFACE": {

             "Loopback0| /32": {}



    Configure the BGP neighbours



             "": {

                "rrclient": "0",

                "name": "OX2",

                "local_addr": "",

                "nhopself": "0",

                "admin_status": "up",

                "holdtime": "180",

                "asn": "65002",

                "keepalive": "60"


            "": {

                "rrclient": "0",

                "name": "CORE1",

                "local_addr": "",

                "nhopself": "0",

                "admin_status": "up",

                "holdtime": "180",

                "asn": "2701",

                "keepalive": "60"


            "": {

                "rrclient": "0",

                "name": "OX1",

                "local_addr": "",

                "nhopself": "0",

                "admin_status": "up",

                "holdtime": "180",

                "asn": "65002",

                "keepalive": "60"


            "": {

                "rrclient": "0",

                "name": "CORE2",

                "local_addr": "",

                "nhopself": "0",

                "admin_status": "up",

                "holdtime": "180",

                "asn": "2701",

                "keepalive": "60"





    The local AS number is part of the DEVCIEMETADATA



    "localhost": {

                "hwsku": "ACS-MSN2100",

                "hostname": "sonic",

                "mac": "24:8a:07:3d:51:00",

                "bgp_asn": "65100",

                "deployment_id": "None",

                "type": "LeafRouter"







    Load the new config file into the RedisDB


    admin@r-stav02:~$ sudo config reload


    Check that BGP is up:


    root@r-stav02:/home/admin#show ip bgp summary

    Command: sudo vtysh -c "show ip bgp summary"

    BGP router identifier, local AS number 65100

    RIB entries 67, using 7504 bytes of memory

    Peers 4, using 18 KiB of memory


    Neighbor        V         AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd        4  2701      18      15        0    0    0 00:10:50       33       4  2701      18      18        0    0    0 00:10:50       33        4 65002      19      18        0    0    0 00:10:48        0        4 65002      18      18        0    0    0 00:10:47        0

    Total number of neighbors 4




    root@r-stav02:/home/admin# show ip route

    Command: sudo vtysh -c "show ip route"

    Codes: K - kernel route, C - connected, S - static, R - RIP,

           O - OSPF, I - IS-IS, B - BGP, P - PIM, A - Babel,

           > - selected route, * - FIB route


    K>* via, eth0

    C>* is directly connected, eth0

    B>* [20/0] via, Ethernet28, src, 00:10:54

      *                     via, Ethernet24, src, 00:10:54

    B>* [20/0] via, Ethernet28, src, 00:10:54

      *                     via, Ethernet24, src, 00:10:54

    B>* [20/0] via, Ethernet28, src, 00:10:54

      *                     via, Ethernet24, src, 00:10:54

    B>* [20/0] via, Ethernet28, src, 00:10:54

      *                     via, Ethernet24, src, 00:10:54




    It is possible to see all of the resources used on the switch


    admin@r-stav02:~$ crm show resources all

    Resource Name           Used Count    Available Count

    --------------------  ------------  -----------------

    ipv4_route                      43             122825

    ipv6_route                       1             122825

    ipv4_nexthop                     4              57342

    ipv6_nexthop                     0              57342

    ipv4_neighbor                    4             122825

    ipv6_neighbor                    0              24576

    nexthop_group_member             2              57342

    nexthop_group                    1              57342

    fdb_entry                        0             122825


    Stage    Bind Point    Resource Name      Used Count    Available Count

    -------  ------------  ---------------  ------------  -----------------

    INGRESS  PORT          acl_group                   0                399

    INGRESS  PORT          acl_table                   0                393

    INGRESS  LAG           acl_group                   0                399

    INGRESS  LAG           acl_table                   0                393

    INGRESS  VLAN          acl_group                   0                399

    INGRESS  VLAN          acl_table                   0                393

    INGRESS  RIF           acl_group                   0                399

    INGRESS  RIF           acl_table                   0                393

    INGRESS  SWITCH        acl_group                   0                399

    INGRESS  SWITCH        acl_table                   0                393

    EGRESS   PORT          acl_group                   0                399

    EGRESS   PORT          acl_table                   1                393

    EGRESS   LAG           acl_group                   0                399

    EGRESS   LAG           acl_table                   0                393

    EGRESS   VLAN          acl_group                   0                  0

    EGRESS   VLAN          acl_table                   0                  0

    EGRESS   RIF           acl_group                   0                399

    EGRESS   RIF           acl_table                   0                393

    EGRESS   SWITCH        acl_group                   0                399


    EGRESS   SWITCH        acl_table                   0                393




    • CRM allows to configure polling interval (in seconds):
    • # crm config polling interval 300
    • CRM allows to configure threshold type:
    • # crm config thresholds ipv4 route type percentage
    • CRM allows to configure threshold values:
    • # crm config thresholds ipv4 route low 70
    • # crm config thresholds ipv4 route high 85

    Syslog message when high threshold exceeded:

    • IPV4_ROUTE THRESHOLD_EXCEEDED for TH_USED 0% Used count 24 free count 57218
    • Syslog message when low threshold exceeded:
    • IPV4_ROUTE THRESHOLD_CLEAR for TH_USED 0% Used count 24 free count 57218


    You can find all the show commands supported in the SONIC CLI:


    admin@r-stav02:~$ show


      aaa                   Show AAA configuration in ConfigDb

      acl                   Show ACL related information

      arp                   Show IP ARP table

      clock                 Show date and time

      ecn                   Show ECN configuration

      environment           Show environmentals (voltages, fans, temps)

      interfaces            Show details of the network interfaces

      ip                    Show IP (IPv4) commands

      ipv6                  Show IPv6 commands

         lldp                  LLDP (Link Layer Discovery Protocol)...

      logging               Show system log

      mac                   Show MAC (FDB) entries

      ntp                   Show NTP information

      platform              Show platform-specific hardware info

      processes             Display process information

      runningconfiguration  Show current running configuration...

      services              Show all daemon services

      session               Show existing everflow sessions

      startupconfiguration  Show startup configuration information

      system-memory         Show memory information

      tacacs                Show TACACS+ configuration

      techsupport           Gather information for troubleshooting

      uptime                Show system uptime

      users                 Show users

      version               Show version information

      vlan                  Show VLAN information



    config commands:


    admin@r-stav02:~$ config


      aaa               AAA command line

      acl               ACL-related configuration tasks

      bgp               BGP-related configuration tasks

      ecn               ECN-related configuration tasks

      interface         Interface-related configuration tasks

      load              Import a previous saved config DB dump file.

      load_mgmt_config  Reconfigure hostname and mgmt interface based...

      load_minigraph    Reconfigure based on minigraph.


      reload            Clear current configuration and import a...

      save              Export current config DB to a file on disk.

      tacacs            TACACS+ server configuration

      vlan              VLAN-related configuration tasks




    use –-help on each config command in order to understand the exact syntax and usage.


    admin@r-stav02:~$ sudo config ecn --help

    Usage: config ecn [OPTIONS]

      ECN-related configuration tasks


      -profile <profile_name>       Profile name  [required]

      -rmax <red threshold max>     Set red max threshold

      -rmin <red threshold min>     Set red min threshold

      -ymax <yellow threshold max>  Set yellow max threshold

      -ymin <yellow threshold min>  Set yellow min threshold

      -gmax <green threshold max>   Set green max threshold

      -gmin <green threshold min>   Set green min threshold



    when using config commands, you will need to execute “config save” in order to apply configuration into the ConfigDB and then “config reload” to update the RedisDB.



    SONiC is based on Containers running its services



    You can see all of the Containers running by executing the command bellow


    admin@r-stav02:~$ docker ps --format 'table {{.Names}}\t{{.Command}}\t{{.Image}}'

    NAMES               COMMAND                  IMAGE

    dhcp_relay          "/usr/bin/docker_init"   docker-dhcp-relay:latest

    snmp                "/usr/bin/supervisord"   docker-snmp-sv2:latest

    syncd               "/usr/bin/supervisord"   docker-syncd-mlnx:latest

    swss                "/usr/bin/supervisord"   docker-orchagent-mlnx:latest

    teamd               "/usr/bin/supervisord"   docker-teamd:latest

    lldp                "/usr/bin/supervisord"   docker-lldp-sv2:latest

    pmon                "/usr/bin/supervisord"   docker-platform-monitor:latest

    bgp                 "/usr/bin/supervisord"   docker-fpm-quagga:latest

    database            "/usr/bin/supervisord"   docker-database:latest



    You can access a container by executing the following command


    admin@r-stav02:~$ docker exec -it bgp bash





    In SONiC, most of the critical network and system data is stored in redisDB.

    DB name

    DB No.




    Application running data



    ASIC configuration and state data



    Counter data for port, lag, queue



    Log level control for SONiC modules



    Source of truth for SONiC configuration



    For PFC watch dog counters control and other plugin extensions



    Configuration state for object in CONFIG_DB


    Checking the RedisDB state can be done as follows:


    root@r-stav02:/home/admin#redis-cli> KEYS INTF*

    1) "INTF_TABLE:Ethernet0:"

    2) "INTF_TABLE:Ethernet4:"

    3) "INTF_TABLE:eth0:"

    4) "INTF_TABLE:docker0:"

    5) "INTF_TABLE:lo:"

    6) "INTF_TABLE:Ethernet28:"

    7) "INTF_TABLE:Ethernet24:"


    more examples can be seen in the following link:



    Splitting a port on SONiC


    The port split feature allows to extend number of available switch ports by splitting one port up to 4 sub-ports (1 per lane).


    Example of supported split options:

    1.   1x100G -> 2x50G

    2.   1x100G -> 4x25G

    3.   1x100G -> 4x10G


    Note: Please review the HW user manual in order of understanding which ports can be split, as an example on the SN2700 only the odd ports can be split, and the adjacent even port should be removed from the port_config.ini


      e.g. if we want to split port Ethernet0 we will need to remove port Ethernet1 from the port_config.ini


    Port split can be configured only in port_config.ini configuration file.

    • SWSS create ports according to the described ports in this file.
    • Ports that are not specified in the file will not be created.


    Example of port_config.ini:


    vim /usr/share/sonic/device/x86_64-mlnx_msn2100-r0/ACS-MSN2100/port_config.ini
    # name         lanes         speed
    Ethernet0      0,1,2,3


    Note: port_config.ini should not contain blank lines because according to current behavior SONiC will throw an error an abort


    Example on how to split port Ethernet0 into 2 ports :

    • Add new port to the list in port_config.ini (e.g. Ethernet2)
    • Reallocate hardware lanes: Ethernet0 – 0,1; Ethernet2 – 2,3
    • Set default speed in example to 50000 (50G)
    • Apply configuration


    Example of “1x100 -> 2x50” split configuration:

    vim /usr/share/sonic/device/x86_64-mlnx_msn2100-r0/ACS-MSN2100/port_config.ini
    # name         lanes         speed
    Ethernet0      0,1           50000
    Ethernet2      2,3           50000


    Note: Speed is mandatory for split ports. If not available, ports will not be created

    Note: The max speed must not exceed 100g for all split ports. On such case ports will not be created


    How to apply changes in port_config.ini?

    There are 2 options,

    1.   Load Minigraph by executing‘config load_minigraph’, which will create new config_db.json based on the Minigraph

    2.   Align config_db json with the changes done in port_config.ini  and reload the configuration.


    Once split is done defaults port’s attributes are used (for example: MTU, speed, admin mode).
    In case non-default values are required need to configure new (sub)ports.
    It can be done like for any other regular not split ports:

    • Through the mingraph.xml
    • In config_db.json configuration file
    • Using Linux CLI commands


    SONiC Telemetry

    Being able to get the underlying characteristics of the network devices - either operational state or configuration, efficiently and quickly,

    will greatly facilitate the analysis of network status and improve network stability.

    Besides the traditional data collecting methods like SNMP, syslog and CLI, gRPC is the modern communication protocol supported by SONiC for telemetry streaming.


    Read more on how to use gRPC with SONiC:





    All SONiC logs are available in /var/log/syslog

    • To change log level use the swssloglevel utility
    • Set orchagent severity level to NOTICE:
      # swssloglevel -l NOTICE -c orchagent
    • Set SAI_API_SWITCH severity to ERROR:
      # swssloglevel -l SAI_LOG_LEVEL_ERROR -s -c SWITCH
    • Set all SAI_API_* severity to DEBUG:
      # swssloglevel -l SAI_LOG_LEVEL_DEBUG -s -a


    Q: What should be done in case the command ‘show interfaces status’ does not show any interfaces.

    A: Ports are created by orchagent process, which is part of SWSS SONiC service.


    1.   Check whether SWSS service is running: # systemctl status swss

    2.   Check whether swss docker container is running: # docker ps

    3.   Check whether orchagent is running in swss docker container: # docker exec -it swss supervisorctl status

    4.   Check /var/log/syslog for errors

    5.   Check port_config.ini file whether it contains valid ports description
    In example for SN2700: /usr/share/sonic/device/x86_64-mlnx_msn2700-r0/ACS-MSN2700/port_config.ini


    Q:What should be done in case some of the docker containers are not running?

    A: The SONiC docker containers are started by SONiC services, so to troubleshoot such a problem you need to:

    • Check whether all services are running: # systemctl status
    • Check /var/log/syslog for errors
    • Restart SONiC service which failed, for example swss: # systemctl restart swss


    Q: How to investigate packet drops?

    A: Packet drops can be investigated by viewing counters using the “show interfaces counters” command

    • RX_DRP
    • TX_DRP




    SONiC Upgrade:


    If you have an old SONiC version, you can easily upgrade it via the SONiC CLI:


    sonic_installer install


    once install is finished you should reboot the switch.


    Join the SONiC community forum !