0 Replies Latest reply on Jan 23, 2018 7:30 AM by agrikk

    MT25418 ConnectX in Ethernet mode in ESXi 6.0?

    agrikk

      I have several MT25418 ("MT25418 [ConnectX VPI - 10GigE / IB DDR, PCIe 2.0 2.5GT/s]") adapters that I am trying to get operational in ESX 6.0 in 10G Ethernet mode.  Right now I'm trying to get them configured point-to-point but I'll probably be adding a 10G Ethernet switch at some future point.

       

      How do I make this card recognizable by ESXi 6.0?

       

      What is the combination of firmware/drivers that will get these adapters to be recognized in ESX and allow me to switch from IB mode to Ethernet mode?

       

      Here's what I have attempted so far:

       

      These HBAs were originally HP-branded HP Infiniband 4X DDR Connect-X PCI-e Dual Port HBA that I flashed to Mellanox-native v2.9.1 firmware on a Windows box.

       

      - Create a fresh install of ESX 6.0

       

      - From the VMware compatibility guide the MT25418 requires the "MLNX Native EN 3.2.0.7 NIC driver", so I download and install the v 3.2.0.7 offline bundle:

       

      esxcli software acceptance set --level=CommunitySupported

      esxcli software vib install -d /tmp/MEL-mlnx-3.2.0.7-offline_bundle-3070178.zip

      reboot

       

       

      After reboot I see:

       

      [root@esx1:~] lspci | grep Mellanox

      0000:04:00.0 Serial bus controller: Mellanox Technologies MT25418 [ConnectX VPI - 10GigE / IB DDR, PCIe 2.0 2.5GT/s] [vmnic2]

      [root@esx1:~]

       

      [root@esx1:~] esxcli network nic list

      Name    PCI Device    Driver  Admin Status  Link Status  Speed  Duplex  MAC Address         MTU  Description

      ------  ------------  ------  ------------  -----------  -----  ------  -----------------  ----  ---------------------------------------------------

      vmnic0  0000:02:00.0  e1000e  Up            Down             0  Half    00:30:48:f6:55:16  1500  Intel Corporation 82574L Gigabit Network Connection

      vmnic1  0000:03:00.0  e1000e  Up            Up            1000  Full    00:30:48:f6:55:17  1500  Intel Corporation 82574L Gigabit Network Connection

       

      [root@esx1:~] esxcli hardware pci list

      0000:04:00.0

         Address: 0000:04:00.0

         Segment: 0x0000

         Bus: 0x04

         Slot: 0x00

         Function: 0x0

         VMkernel Name: vmnic2

         Vendor Name: Mellanox Technologies

         Device Name: MT25418 [ConnectX VPI - 10GigE / IB DDR, PCIe 2.0 2.5GT/s]

         Configured Owner: Unknown

         Current Owner: VMkernel

         Vendor ID: 0x15b3

         Device ID: 0x634a

         SubVendor ID: 0x15b3

         SubDevice ID: 0x0006

         Device Class: 0x0c06

         Device Class Name: InfiniBand

         Programming Interface: 0x00

         Revision ID: 0xa0

         Interrupt Line: 0x0a

         IRQ: 10

         Interrupt Vector: 0x2e

         PCI Pin: 0x00

         Spawned Bus: 0x00

         Flags: 0x0201

         Module ID: 4109

         Module Name: vmklinux_9

         Chassis: 0

         Physical Slot: 6

         Slot Description: Chassis slot 6; function 0

         Passthru Capable: false

         Parent Device: PCI 0:0:2:0

         Dependent Device: PCI 0:4:0:0

         Reset Method: Function reset

         FPT Sharable: false

       

      [root@esx1:~] esxcli system module list | grep mlx4

      nmlx4_core                          true        true

      mlx4_core                           true        true

      mlx4_en                             true        true

      nmlx4_en                           false        true

      [root@esx1:~]

       

       

      As you can see, the nmlx4_en Mellanox Ethernet module isn't loading. And I suspect it is due to the card defaulting to IB mode somehow somewhere.

       

      How do I make this card recognizable by ESXi 6.0?