2 Replies Latest reply on Nov 20, 2018 1:25 AM by samerka

    Unable to set Mellanox ConnectX-3 to Ethernet (Failed to query device current configuration)

    victorhooi

      I have three Mellanox ConnectX-3 cards, that I'm trying to setup with Proxmox (Proxmox Installer does not see Mellanox ConnectX-3 card at all? | Proxmox Support Forum)

      I need to change them from Infiniband mode to Ethernet mode.

      I was able to install the Mellanox Management Tools, and they can see my card:

      root@gcc-proxmox:~/mft-4.10.0-104-x86_64-deb# mst start
      Starting MST (Mellanox Software Tools) driver set
      Loading MST PCI module - Success
      Loading MST PCI configuration module - Success
      Create devices
      root@gcc-proxmox:~/mft-4.10.0-104-x86_64-deb# mst status
      MST modules:
      ------------
        MST PCI module loaded
        MST PCI configuration module loaded
      
      MST devices:
      ------------
      /dev/mst/mt4099_pciconf0 - PCI configuration cycles access.
        domain:bus:dev.fn=0000:41:00.0 addr.reg=88 data.reg=92
        Chip revision is: 01
      /dev/mst/mt4099_pci_cr0 - PCI direct access.
        domain:bus:dev.fn=0000:41:00.0 bar=0xd4f00000 size=0x100000
        Chip revision is: 01
      

      However, when I tried to query the current config, it complains about the firmware version being too old.

      root@gcc-proxmox:~/mft-4.10.0-104-x86_64-deb# mlxconfig -d /dev/mst/mt4099_pciconf0 q -E- Failed to open device: /dev/mst/mt4099_pciconf0. Unsupported FW (version 2.31.5000 or above required for CX3/PRO)
      

      So I updated the firmware:

      root@gcc-proxmox:~# flint -d /dev/mst/mt4099_pci_cr0 -i fw-ConnectX3-rel-2_42_5000-MCX311A-XCA_Ax-FlexBoot-3.4.752.bin burn
      
      
          Current FW version on flash:  2.10.4290
          New FW version:               2.42.5000
      
      
      Burn process will not be failsafe. No checks will be performed.
      ALL flash, including the Invariant Sector will be overwritten.
      If this process fails, computer may remain in an inoperable state.
      
      
       Do you want to continue ? (y/n) [n] : y
      Burning FS2 FW image without signatures - OK
      Restoring signature                     - OK
      

      But now when I try to read the config - I get a new error:

      root@gcc-proxmox:~# mlxconfig -d /dev/mst/mt4099_pciconf0 q
      
      
      Device #1:
      ----------
      
      
      Device type:    ConnectX3
      Device:         /dev/mst/mt4099_pciconf0
      
      
      Configurations:                              Next Boot
      -E- Failed to query device current configuration
      

      Any ideas what's going on, or how to get these cards working in permanent Ethernet mode?