5 Replies Latest reply on Aug 16, 2016 11:47 AM by inbusiness

    VMware ESXi 6.0 virtual ib_ipoib interfaces

    ciscokid

      Newbie question:  What is the correct syntax (if at all possible with version of VMware ESXi 6.0 U1 and OFED 2.4.0.0) for creating sub/virtual interfaces?  From what I can gather in the documentation would it be something like the following:

       

      esxcli system module parameters set -m mlx4_core -p set_4k_mtu=1,msi_x=1,num_vfs=16,probe_vf=16

       

      After rebooting I have not be able to see any additional interfaces prompting this query on the forum.  Please advise.

        • Re: VMware ESXi 6.0 virtual ib_ipoib interfaces
          sophie

          Hi david,

          Can you confirm what type of HCA card(s) installed within the ESXI server?

          What is the port protocol configured as IB or Ethernet?

           

          Thank you,

          Sophie.

            • Re: VMware ESXi 6.0 virtual ib_ipoib interfaces
              joachim

              I have a similar problem: i cant get IP over IB working.

              MT 27500 ConnectX-3 (PSID MT_1100120019) on ESX 6 (latest Updates 2 days ago). In the BIOS (SuperMicro) I set in the PCIe Configuration Page “Above 4G Decoding” as “Disabled” since otherwise the card was not recognized in older SuperMicro hardware.


              First installed the 3.15.5.5 and also updated the firmware to 5150

              Then had a vmnic2 with up to 40000, but link was down.


              Then I uninstalled and installed OFED 2.4.0.0
              Then had a vmnic2 with up to 1000GB, link down.

               

              I also tried to install both drivers, same issue: link down, card is recognized as Ethernet and not IPoIB.
              I already opened a support ticket with VMWare but the technician had no solution.

               

              It seems the last know driver to work is 1.8.2.4 (see here VSphere 6.0 ) but this driver is not supported in 6.x
              By the way: when you select now OFED 2.4.0.0. you are redirected to 2.3.3.1 for download (http://www.mellanox.com/page/products_dyn?product_family=36&mtag=vmware_drivers )

               

              I had no problem with the same cards in similar (older) hardware and ESX 5.5.

            • Re: VMware ESXi 6.0 virtual ib_ipoib interfaces
              sophie

              Hi David,

               

              1) Driver version 2.4.0 is an Ethernet driver only so if you have a VPI card, please make sure the port protocol is configured as 2 which is Ethernet.

               

              #/opt/mellanox/bin/mlxconfig -d /dev/mt4099_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

               

              2) Enable SRIOV with the MFT tool (Mellanox FW tool) as follow:

               

              #/opt/mellanox/bin/mlxconfig -d /dev/mt4099_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=16

               

              3) Confirm BIOS setting supports SRIOV and/or virtualization.

               

              4) Set the driver module parameters for VF:

              #esxcli system module parameters set -m mlx4_core -p 'num_vfs=<VFs over Port1, VFs over port2, 0> port_type_array=2'

               

              This is all documented in the driver UM.

               

              Thank you,

              Sophie.