0 Replies Latest reply on Dec 3, 2018 2:35 PM by

    Virtual functions are not created on SRIOV setup with ConnectX-3 and Vmware ESXi 6.0

      Hi Team,

      I am trying to setup SRIOV with ConnectX-3 card using Vmware ESXi 6.0.  I have followed these documents on the Mellanox website,

      http://www.mellanox.com/related-docs/MFT/MFT_user_manual_4_10_0.pdf

      http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_ESXi_User_Manual_v2.4.0.pdf

      to setup SRIOV.  I have installed the necessary via files, configured the ports for Ethernet,  configured device for SRIOV_EN and NUM_OF_VFS but I don’t see the virtual functions when I execute lspci |grep Mellanox.  Can you please advice. Thanks.

       

      /opt/mellanox/bin/mlxconfig -d mt4099_pciconf0 set SRIOV_EN=1 NUM_OF_VFS=16

      /opt/mellanox/bin/mlxconfig -d mt4099_pciconf0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2

      esxcli system module parameters set -m mlx4_core -p 'num_vfs=8 port_type_array=2'

       

      [kajens1@host1:/var/log] esxcli software vib list | grep MEL

      net-ib-core                    2.4.0.0-1OEM.600.0.0.2494585           MEL          PartnerSupported  2018-09-17

      net-ib-ipoib                   2.4.0.0-1OEM.600.0.0.2494585           MEL          PartnerSupported  2018-09-17

      net-ib-mad                     2.4.0.0-1OEM.600.0.0.2494585           MEL          PartnerSupported  2018-09-17

      net-ib-sa                      2.4.0.0-1OEM.600.0.0.2494585           MEL          PartnerSupported  2018-09-17

      net-mlx-compat                 2.4.0.0-1OEM.600.0.0.2494585           MEL          PartnerSupported  2018-09-17

      net-mlx4-core                  2.4.0.0-1OEM.600.0.0.2494585           MEL          PartnerSupported  2018-09-17

      net-mlx4-en                    2.4.0.0-1OEM.600.0.0.2494585           MEL          PartnerSupported  2018-09-17

      net-mlx4-ib                    2.4.0.0-1OEM.600.0.0.2494585           MEL          PartnerSupported  2018-09-17

      nmlx4-core                     3.15.5.5-1OEM.600.0.0.2768847          MEL          VMwareCertified   2018-05-06

      nmlx4-en                       3.15.5.5-1OEM.600.0.0.2768847          MEL          VMwareCertified   2018-05-06

      nmlx4-rdma                     3.15.5.5-1OEM.600.0.0.2768847          MEL          VMwareCertified   2018-05-06

      nmlx5-core                     4.15.8.8-1OEM.600.0.0.2768847          MEL          VMwareCertified   2018-05-06

      nmst                           4.11.0.103-1OEM.600.0.0.2768847        MEL          PartnerSupported  2018-12-03

       

       

      [kajens1@host1:/var/log] esxcli software vib list | grep mft

      mft                            4.11.0.103-0                           Mellanox     PartnerSupported  2018-12-03

       

       

      [kajens1@host1:/var/log] mst status

      MST devices:

      ------------

      mt4099_pciconf0

      mt4099_pci_cr0

       

       

      [kajens1@host1:/var/log] mst status -vv

      PCI devices:

      ------------

      DEVICE_TYPE             MST                           PCI       RDMA            NET                       NUMA

      ConnectX3(rev:1)        mt4099_pciconf0               01:00.0                   net-vmnic5,net-vmnic1000502

      ConnectX3(rev:1)        mt4099_pci_cr0                01:00.0                   net-vmnic5,net-vmnic1000502

       

       

      [kajens1@host1:/var/log] esxcli network nic list

      Name          PCI Device    Driver     Admin Status  Link Status  Speed  Duplex  MAC Address         MTU  Description

      ------------  ------------  ---------  ------------  -----------  -----  ------  -----------------  ----  -------------------------------------------------------

      vmnic0        0000:1b:00.0  tg3        Up            Up            1000  Full    08:94:ef:06:fa:b8  1500  Broadcom Corporation NetXtreme BCM5719 Gigabit Ethernet

      vmnic1000502  0000:01:00.0  nmlx4_en   Up            Up           40000  Full    00:02:c9:35:8f:c1  1500  Mellanox Technologies MT27500 Family [ConnectX-3]

      vmnic5        0000:01:00.0  nmlx4_en   Up            Up           40000  Full    00:02:c9:35:8f:c0  1500  Mellanox Technologies MT27500 Family [ConnectX-3]

       

       

      [kajens1@host1:/var/log] mlxconfig -d mt4099_pciconf0 q

       

      Device #1:

      ----------

       

      Device type:    ConnectX3

      Device:         mt4099_pciconf0

       

      Configurations:                              Next Boot

               SRIOV_EN                            True(1)

               NUM_OF_VFS                          16

               LINK_TYPE_P1                        ETH(2)

               LINK_TYPE_P2                        ETH(2)

               LOG_BAR_SIZE                        3

               BOOT_PKEY_P1                        0

               BOOT_PKEY_P2                        0

               BOOT_OPTION_ROM_EN_P1               True(1)

               BOOT_VLAN_EN_P1                     False(0)

               BOOT_RETRY_CNT_P1                   0

               LEGACY_BOOT_PROTOCOL_P1             PXE(1)

               BOOT_VLAN_P1                        1

               BOOT_OPTION_ROM_EN_P2               True(1)

               BOOT_VLAN_EN_P2                     False(0)

               BOOT_RETRY_CNT_P2                   0

               LEGACY_BOOT_PROTOCOL_P2             PXE(1)

               BOOT_VLAN_P2                        1

               IP_VER_P1                           IPv4(0)

               IP_VER_P2                           IPv4(0)

               CQ_TIMESTAMP                        True(1)

       

       

      [kajens1@host1:/var/log] mlxfwmanager --query

      Querying Mellanox devices firmware ...

       

      Device #1:

      ----------

       

        Device Type:      ConnectX3

        Part Number:      MCX354A-FCB_A2-A5

        Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6

        PSID:             MT_1090120019

        PCI Device Name:  mt4099_pci_cr0

        Port1 MAC:        0002c9358fc0

        Port2 MAC:        0002c9358fc1

        Versions:         Current        Available

           FW             2.42.5000      N/A

           PXE            3.4.0752       N/A

       

        Status:           No matching image found

       

       

      [kajens1@host1:/var/log] lspci |grep Mellanox

      0000:01:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] [vmnic5]

       

      Thanks,

      Kajen Siva