13 Replies Latest reply on Feb 2, 2014 6:46 AM by yairi

    Mellanox Voltaire 4036 VLT-3011s way to reflash OS / reset password

      Hi,

       

      I bought two Mellanox Voltaire 4036 VLT-3011 switches from EBAY.

       

      These seem both to have issues. I need help diagnosing and fixing these if possible.

       

      I have serial console cable and it works with both units. I'm able to get to the

      login prompts and see U-boot options and access U-boot boot menu,

       

      But I can't get pass the login prompts in both units.

       

      The OS versions are below:

       

      1. 4036 Version 3.9.1 BUILD ID 985

      2. 4036 Version 3.6.2 BUILD ID 872

       

      1- The first (1) unit, does not let me log on as admin/123456 or root/br6000 or guest/voltaire

       

      It accepts the admin and guest logons but immediately after login prompt it logs the user out,

      and I do not get to the OS level at all.

       

      I found during the boot process the following warnings:

       

      Empty flash at 0x02eb8908 ends at 0x02eb9000:

      Empty flash at 0x0613c8e8 ends at 0x0613d000

       

      I get immediately the following warning:

      ibwarn: [896] mad_rpc_open_port: can't open UMAD port ((null):0)

       

      and user is logged out, and getty terminal is reset

       

      Does this mean there is no OS on the flash as its empty and I can not login ??

       

      Can this switch even be used as IB switch?

       

       

      This unit lights up all the switch port leds and they are on all the time !!

       

      2. The second (2) unit does not accept any admin/guest/root password and complains

      about invalid login/password.

       

      I have tried reset command from U-boot menu and also pressed the reset switch on the Voltaire 4036

      for 30 seconds and the switches reboot but do not reset the password on this unit.

       

      This unit does not light up the switch port leds. But they blink once during the reboot.

       

       

      My questions:

       

      - Can I reset the password somehow on the 2) unit that I can not login into ?

      - Can I reload OS on the 1) unit which complains about Empty Flash and I cant login as admin ?

      - Can these switches be used as IB switches but without management features ?

      - Can the SM tools spark etc. be used to reflash firmware remotely  ?

       

      @yairi and justinclift been helping others with upgrade/flashing/reset procedures for 4036 and isr 9024d-m

       

      Can you @yairi and justinclift help me here to analyse the situation with these 2 4036 switches

      and what can I do to gain access and start using the switches or are they dead bricks?

       

      Also inbusiness might be interested to help

       

       

      As there is U-boot, I've seen TFTP option and   flash_NFS

       

      - tftpboot- boot image via network using TFTP protocol

      - Type run flash_nfs to mount root filesystem over NFS

       

      Can I use above to load image via TFTP on U-Boot or flash_nfs for root filesystem ?

       

      I haven't found any instructions or anyone succeeding in doing this. Can we try to work this out ?

       

      I've read these:

      Upgrading 4036, media full

      voltaire 4036 cant connect to CLI

      Mellanox (old Voltaire) ISR9024D-M recover flash area

       

       

      Here somebody has tried TFTP / NFS but fails:

      http://forums.servethehome.com/networking/2060-voltaire-mellanox-4036-firmware-update-2.html

       

      I've attached some logfiles from the terminal sessions from both units as well as the uboot prompt menu commands.

       

      Thanks for any suggestions

        • Re: Mellanox Voltaire 4036 VLT-3011s way to reflash OS / reset password
          justinclift

          Personally, I no longer help out on these forums nor advise people to buy Mellanox gear, due to their phishing-like emails (which they've been warned about).

           

          Good luck with your problem though.

          • Re: Mellanox Voltaire 4036 VLT-3011s way to reflash OS / reset password
            yairi

            Hi,

             

            generally speaking, these switches can do the IB native switching operation even if the SW is in bad state. unless their is something wrong with the switch ASIC or the software/HW (PCI) keeps on resetting the ASIC.

             

            About your "first switch", the one with all lights turned on - to me it looks like it is stack in some sort of reset sequence. what we want to try is the following:

            • Open the box and check if everything is connected. no loos components, wires, etc. if this is something you haven't done before, you might want to call somebody up to do it with you.
            • set it back to factory defaults: since you have no access to CLI or OS, there is a way to do it from the uboot menu.
              • connect a console cable and power up the switch and immidiately after it starts, stop the boot sequence (^c or any key)
              • the uboot prompt should look like this "=>"
              • run the following commands:
                • imm 53 90.1

                • 01

                • enter

                • ctr+c

                • boot

              • wait until the boot sequence is complete and you received back the login prompt. login with default user/pass (those you mentioned are fine).

            Let's hope that the HW is in good shape and this is helping. other then that, i don't think there is anything else that can be done.

             

            About your "second switch": The LED sequence is correct so it is a good sign. somebody probably set it up with a password. go through the same procedure to factory reset it as above and start with a brand new configuration. later on i suggest upgrading to the latest version 3.9.1.

             

            Good luck.

              • Re: Mellanox Voltaire 4036 VLT-3011s way to reflash OS / reset password

                yairi

                I tried the password reset function uboot menu and it worked, but the first unit logs

                me out of the console after password change and even if I try to relogin it kills the login process

                so something is wrong with the OS ?

                 

                 

                the second unit it now lets me logon to the console an it seems to be working properly.

                 

                 

                So thanks for the help so far.

                 

                Is there any way for me to reinstall the OS on the first unit ?

                 

                Here's a log of what is happening on the first unit during password change and logon:

                 

                4036 Version 3.9.1 BUILD ID 985

                Sun Sep 23 18:01:19 IST 2012

                4036-28C4 login: admin

                Password:

                 

                Welcome to Voltaire Switch 4036-28C4

                 

                Initial configuration: Please change the default root password

                Changing password for root

                Enter the new password (minimum of 5, maximum of 8 characters)

                Please use a combination of upper and lower case letters and numbers.

                Enter new password:

                Re-enter new password:

                Password changed.

                ibwarn: [848] mad_rpc_open_port: can't open UMAD port ((null):0)

                process '-/sbind

                4036 Version 3.9.1 BUILD ID 985

                Sun Sep 23 18:01:19 IST 2012

                4036-28C4 login:

                 

                 

                Also during kernel load I saw these messages on first unit:

                 

                Loading module Voltaire

                Empty flash at 0x02eb8908 ends at 0x02eb9000

                Empty flash at 0x0613c8e8 ends at 0x0613d000

                Empty flash at 0x08d1d504 ends at 0x08d1d800

                 

                 

                 

                These are not present in the working (second) unit.

                • Re: Mellanox Voltaire 4036 VLT-3011s way to reflash OS / reset password

                  First unit - I checked inside of the unit and there is no loose wires or anything.

                   

                  All the port leds are lit on the system and I get the Empty Flash during reboot process

                  and when I try to login it logs me out as it cant run the shell and I get a warning:

                   

                   

                   

                  yairi

                  I tried the password reset function uboot menu and it worked, but the first unit logs

                  me out of the console after password change and even if I try to relogin it kills the login process

                  so something is wrong with the OS ?

                   

                   

                  the second unit it now lets me logon to the console an it seems to be working properly.

                   

                   

                  So thanks for the help so far.

                   

                  Is there any way for me to reinstall the OS on the first unit ?

                   

                  Here's a log of what is happening on the first unit during password change and logon:

                   

                   

                  ibwarn: [848] mad_rpc_open_port: can't open UMAD port ((null):0)

                  process '-/sbind

                    • Re: Mellanox Voltaire 4036 VLT-3011s way to reflash OS / reset password
                      yairi

                      Hi Joe,

                      I wouldn't get my hopes high on the first unit.

                      I might be wrong but this is what I am thinking: the fact that all LEDs are on is probably because the switch ASIC is stack in reset.

                      the UMAD port 0 error means that the OS can't send management packets to the ASIC.

                      the empty flash errors may be related to the ASIC flash area. this is where the switch ASIC stores the FW. maybe this part is bad.

                       

                      all together, i think you have a bad switch here. i am not sure if you will be able to recover it beyond what we tried.

                       

                      Sorry i can't help further more. hope you enjoy you other switch unit :-)

                  • Re: Mellanox Voltaire 4036 VLT-3011s way to reflash OS / reset password
                    yairi

                    No worries. i am still working with my team on that.

                     

                    Thanks!

                    • Re: Mellanox Voltaire 4036 VLT-3011s way to reflash OS / reset password

                      Thanks for your help.

                       

                      I returned the defective item.

                      They are offering another one in exchange or full refund.

                       

                      However, there is a red led inside the case  of the  other unit blinking/turned on.

                       

                      Is that alarming ? do you know ?  Is it in error condition ?

                      Or just a power/status led or is it showing that no network connectivity?

                       

                      Can I take it ? They can only do power on testing, and not any other kind of tests.

                       

                      Otherwise only the PSU led is turned on this another unit.