6 Replies Latest reply on Mar 17, 2018 4:07 PM by agrikk

    Infiniband HBA ports show as "cable unplugged" in Windows 2016?

    agrikk

      I've posted this topic on other forums on the internet with mediocre results, so hopefully coming straight to the source will help me resolve this:

       

      I am setting up a new IB storage network for my home lab and I need help establishing a connection with my Windows Server 2016 box.

       

      The hardware is:

      HP DL180 G6 server with Windows Server 2016 x64

      HP Infiniband 4X DDR Connect-X PCI-e Dual Port HBA (Part number is 448397-B21 / 452372-001)

      Infiniband SFF-8740 cable

      Topspin 120 24-port IB switch

       

      I have upgraded the firmware on the Topspin switch to 2.9.0/build 170 and have installed the Mellanox WinOF VPI (v5.35) on my server. I have set the MTU on both ports of the HBA to 2048 to match the switch settings.

       

      The card detects as a Mellanox ConnectX interface card in the OS, but the network adapter appears as "cable unplugged" for both ports. I have used WinOF to configure the ports as IPoIB and as Ethernet just to see if one works over the other, but both are down. Cabling the card do the same card in another Windows box and setting both to Ethernet mode brings up the connection immediately, so the issue is definitely Infiniband related.

       

      mst status shows:

      mt25418_pci_cr0

      mt25418_pciconf0

       

      I have flashed the cards from HP to Mellanox firmware 2.9.1 using the following command:

       

      flint -d mt25418_pciconf0 -i fw-25408-2_9_1000-MHGH28-XTC_A2-A3.bin -nofs -allow_psid_change burn

       

       

      There is a link light on the switch for both connections as well as a link light on the HBA on both ports. The ports on the switch show Up/Up. The ports are set to auto-negotiate and show up at 4x-SDR(10Gbps). When I connect the adapters to the Topspin 120 switch and put them into IPoIB mode, get get a good green light on the adapter and switch, and the amber light is lit on each adapter, indicating a good connection.

       

      In addition, the switch itself shows the two servers as up and active nodes in the IB section of the management console, and the IB HBAs show up as active ports and functional neighbors. So the subnet manager on the switch is functioning as it should, I guess.

       

      So everything seems good and all, but the HBAs still show as Network Cable unplugged within Windows and I am not able to send any traffic through them.

       

       

      What have I missed? How come the adapters show "cable unplugged" in the OS?

        • Re: Infiniband HBA ports show as "cable unplugged" in Windows 2016?
          aviap

          Will start with indicating the fact that Mellanox WinOF VPI (v5.35) provides modules that are containing IB & ETH drivers supporting ConnectX-3 & ConnectX-3 pro adapters only, and over FW versions 2.40.5032, so inspire upgrading switch fw & getting "green line" at the switch ports it seems to me that you may have an issue with SM (subnet manager) on the switch or incompatibility between the operating-system tack and SM. Suggesting that you liaise with Cisco to check if they can provide a compatibility matrix that confirms workability of all components above (Mellanox Connect-X PCI-e Dual Port adapter and the proper fw, IPoIB, switch fw etc...)