5 Replies Latest reply on Jan 16, 2018 7:45 AM by alltheasmiov

    Questions about IS5031 IB switch licenses and cards

    alltheasmiov

      I purchased a used IS5030 switch, serial number: 98Y3580YM10AG3AA009.  I think it’s actually an IS5031 since only 18 of the ports are enabled, but the IS5031’s all seem to be labeled IS5030. I was able to login with the console and with the WebUI. No licenses are installed, so I can’t start the internal subnet manager. I made a test setup involving the switch, two nodes with Qlogic QLE7340 cards, and QDR cables. By running opensm on one of the nodes, I was able to test ports 1-18 successfully with ibping. So far so good.

       

      Unfortunately, all of the links in the manuals are dead. Here is what I need to help with:

      • Where can I obtain a license for using the internal subnet manager? The link in the manual is dead.
      • Where can I obtain a license to enable all 36 ports? The link in the manual is dead.
      • Where can I find this switch’s firmware so I can upgrade it? It currently has “EFM_PPC_M405EX EFM_1.1.2500 2011-02-22 16:13:31 ppc” FabricIT installed, and firmware version “7.4.2020”. I tried searching on mellanox.com firmware, but I couldn't find the IS5030 or IS5031.
      • One possibly not-mellanox question: Why are the QLE7340 cards negotiating to 20 Gb/s rate (shown in ibstat)? Lspci –vv gives “LnkSta: Speed 5Gt/s, Width x8” . The QLE7340 is pcie gen2, the PCI slot it is in is gen3 x8, so it should be able to do full QDR. In the switch’s webui, under ib0 state, it says that the speed is “40Gb/s (4x QDR)”. So why is it only 20Gb/s? I have a Sun switch and some HP cards that I've confirmed work at 40Gb/s, so I'm going to try the QLE7340 cards with the Sun IB switch, and try the HP cards with the IS5031 switch to figure who the culprit is.

       

      Thank you for any help. I will write up a detailed guide on what I find so that some of these older switches can find loving hobbyist homes.

       

      Edit: More info: The nodes are all running Centos7 with latest updates and "Infiniband Support" package installed. I install opensm on the node that runs opensm.

       

        • Re: Questions about IS5031 IB switch licenses and cards
          alltheasmiov

          I did the tests mentioned in the fourth bullet. With confirmed QDR working IB cards and cables and the IS5031, the cards negotiated to 20 Gb/s rate, so it must be something with the switch. I also tried the QLE7340 with the Sun switch (which I've confirmed works at 40 Gb/s). Interestingly, the QLE70340 negotiated to 10 Gb/s rate. So something is screwy with both the switch and with the QLE cards. Maybe something to do with firmware mismatching?

            • Re: Questions about IS5031 IB switch licenses and cards
              alltheasmiov

              I tried two QLE7340's with the Sun switch, and they both negotiated to 10 Gb/s. They negotiate to 20 Gb/s with the IS5031. Summary of tests:

              • Two QLE7340, IS5031 switch, opensm : both negotiate to 20 Gb/s
              • HP rebranded mellanox HCAs, Sun switch with internal SM : 40 Gb/s
              • HP rebranded mellanox HCAs, IS5031 switch, opensm : 20 Gb/s
              • QLE7340 and HP rebranded mellanox HCA, Sun switch with internal SM : 40 Gb/s for HP card, 10 Gb/s for QLE7340
              • Two QLE7340, Sun switch with internal SM : both 10 Gb/s

               

              I did a lot of googling. I've learned a little about PSM vs. verbs. Qlogic/Intel Infinipath uses PSM, and Mellanox/everyone else uses verbs. I know that when using MPI, you have to use one or the other, and that using Qlogic/Intel HCAs with verbs is slower. I've read some posts that say that you can't mix qlogic and other brands, and some posts that say you can. But I don't really understand it. Anyone have any insight on this?

               

              I also learned that the Qlogic HCAs don't have firmware and that you're supposed to use the True scale fabric suite (OFED+ is the free version I guess?) from Intel. I'll give that a try next.

               

              Still haven't found anything about the two licenses or the firmware for the IS5031. I sent a support inquiry...I don't have a warranty or anything, but we'll see.

                • Re: Questions about IS5031 IB switch licenses and cards
                  alltheasmiov

                  Progress! I contacted Mellanox support through their support website. Very fast response.

                   

                  If your switch was sold after 2012, it should have the FabricIT/Subnet Manager license on the underside of the pull out tab. If it was not, you need to contact technical support with your switch's serial number and they can check for it internally.

                   

                  They said that when you upgrade FabricIT, which runs the subnet manager, that the firmware also gets upgraded automatically. The last FabricIT version (you can see your switch's version using the "show version" command) is 1.1.3004, and the hardware is end of life, so if you have that version then you don't need to do any upgrades. Mine is 1.1.2500, so I need to upgrade it.

                   

                  Process for upgrading FabricIT software and firmware of IS5030 or IS5031 IB switches:

                  1. Download latest firmware image. This is the link they gave me, but it might be temporary. Here it is on my google drive.
                  2. Starting from Section 5.1, follow the instructions in “FabricIT Enterprise Fabric Management Software User Manual For EFM Rev 1.1.3004”. (This is the link they gave me, link on my google drive.). This assumes you can already ssh into the switch. If not, read the Installation Guide (available online) or earlier sections of that User Manual for how to setup the ethernet port.
                  3. I had to modify the given steps slightly. Note that I configured my switch with a static IP of 192.168.0.3. Here are my modifications:

                   

                  1. Since I didn’t set up a user or password, the ssh login was “ssh -l admin 192.168.0.3”.
                  2. I didn’t have any images available to be installed, so I skipped the deletion step.
                  3. If you are using a modern linux system to ssh into the switch in order to scp from the switch, you will receive an error “no kex alg”. The is because some of the kex algorithms have been found to have security vulnerabilities since the time the switch was made, and so have been excluded. You need exit out of the switch ssh, then append the following line to your /etc/ssh/sshd_config: “KexAlgorithms diffie-hellman-group1-sha1,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1” (without quotes). Then restart sshd. Then log back into the switch. You should remove that line when you’re done with scp and restart sshd.
                  4. My scp line looked like “image fetch scp://<Linux username>@192.168.0.1/home/<Linux username>/Downloads/image-PPC_M405EX-EFM_1.1.3004.img”, where the IP address is the IP address of your computer's ethernet adapter. It then prompted me for a password and I input my Linux account’s password. This worked: "show images" showed an image ready to be installed.
                  5. I wasn’t prompted to save configuration.
                  6. After reload, I had to wait about 5 minutes in order to ssh back into it. After running “show version”, the new firmware was present.
                  7. All other steps were the same as in the User Manual.

                   

                  As for the upgrade to 36 ports license, they directed me to sales@mellanox.com. I sent them an email, so we'll see... Also still waiting on the subnet manager license. The HCAs that were confirmed to work at QDR with the Sun switch are still only operating at 20 Gb/s with the IS5031 switch with opensm running on one of the nodes.