HowTo Install Mellanox OFED ESX Driver for VMWare ESX 5.5

Version 6

    This is a basic post that shows how to install MLNX-OFED ESX Driver on ESX 5.5 VMWare host, and perform basic initial configuration steps to enable the driver.

     

    References

     

    Driver Installation

    1. Verify that the host is equipped with Mellanox adapter

     

    ~ # lspci | grep Mellanox

    0000:02:00.0 Network controller: Mellanox Technologies MT27500 Family [ConnectX-3] [vmnic0]

    ~ # esxcli network nic list

    Name          PCI Device     Driver   Link  Speed  Duplex  MAC Address         MTU  Description

    ------------  -------------  -------  ----  -----  ------  -----------------  ----  ---------------------------------------------------

    vmnic0        0000:002:00.0  mlx4_en  Up    40000  Full    00:02:c9:a0:9a:10  1500  Mellanox Technologies MT27500 Family [ConnectX-3]

    vmnic1        0000:005:00.0  e1000e   Up     1000  Full    00:25:90:4f:ea:fa  1500  Intel Corporation 82574L Gigabit Network Connection

    vmnic1000002  0000:002:00.0  mlx4_en  Down      0  Half    00:02:c9:a0:9a:11  1500  Mellanox Technologies MT27500 Family [ConnectX-3]

    vmnic2        0000:006:00.0  e1000e   Down      0  Half    00:25:90:4f:ea:fb  1500  Intel Corporation 82574L Gigabit Network Connection

    ~ #

     

    Note: in this case, Mellanox card is using vmnic0.

     

    2. Use SCP or any other file transfer method to copy the driver to the required ESX host.

     

    3. Install the ESXi Driver:

     

    ~ # esxcli software vib install -d /tmp/MLNX-OFED-ESX-1.9.10.0-10EM-550.0.0.1331820.zip

    Installation Result

       Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.

       Reboot Required: true

       VIBs Installed: Mellanox_bootbank_net-ib-addr_1.9.10.0-1OEM.550.0.0.1331820, Mellanox_bootbank_net-ib-cm_1.9.10.0-1OEM.550.0.0.1331820, Mellanox_bootbank_net-ib-core_1.9.10.0-1OEM.550.0.0.1331820, Mellanox_bootbank_net-ib-mad_1.9.10.0-1OEM.550.0.0.1331820, Mellanox_bootbank_net-ib-sa_1.9.10.0-1OEM.550.0.0.1331820, Mellanox_bootbank_net-ib-umad_1.9.10.0-1OEM.550.0.0.1331820, Mellanox_bootbank_net-mlx4-core_1.9.10.0-1OEM.550.0.0.1331820, Mellanox_bootbank_net-mlx4-en_1.9.10.0-1OEM.550.0.0.1331820, Mellanox_bootbank_net-mlx4-ib_1.9.10.0-1OEM.550.0.0.1331820, Mellanox_bootbank_net-rdma-cm_1.9.10.0-1OEM.550.0.0.1331820, Mellanox_bootbank_scsi-ib-iser_1.9.10.0-1OEM.550.0.0.1331820

       VIBs Removed:

       VIBs Skipped:

    ~ #

     

    4. Reboot the server

    #reboot

     

    5. Check that the driver modules are loaded:

    ~ # esxcli software vib list | grep Mellanox

    net-ib-addr                    1.9.10.0-1OEM.550.0.0.1331820         Mellanox  PartnerSupported  2015-03-13

    net-ib-cm                      1.9.10.0-1OEM.550.0.0.1331820         Mellanox  PartnerSupported  2015-03-13

    net-ib-core                    1.9.10.0-1OEM.550.0.0.1331820         Mellanox  PartnerSupported  2015-03-13

    net-ib-mad                     1.9.10.0-1OEM.550.0.0.1331820         Mellanox  PartnerSupported  2015-03-13

    net-ib-sa                      1.9.10.0-1OEM.550.0.0.1331820         Mellanox  PartnerSupported  2015-03-13

    net-ib-umad                    1.9.10.0-1OEM.550.0.0.1331820         Mellanox  PartnerSupported  2015-03-13

    net-mlx4-core                  1.9.10.0-1OEM.550.0.0.1331820         Mellanox  PartnerSupported  2015-03-13

    net-mlx4-en                    1.9.10.0-1OEM.550.0.0.1331820         Mellanox  PartnerSupported  2015-03-13

    net-mlx4-ib                    1.9.10.0-1OEM.550.0.0.1331820         Mellanox  PartnerSupported  2015-03-13

    net-rdma-cm                    1.9.10.0-1OEM.550.0.0.1331820         Mellanox  PartnerSupported  2015-03-13

    scsi-ib-iser                   1.9.10.0-1OEM.550.0.0.1331820         Mellanox  PartnerSupported  2015-03-13

    ~ # esxcli system module list | grep mlx4

    mlx4_core                           true        true

    mlx4_en                             true        true

    mlx4_ib                             true        true

    ~ # esxcli system module list | grep ib_

    ib_core                             true        true

    ib_mad                              true        true

    ib_sa                               true        true

    ib_umad                             true        true

    ib_cm                               true        true

    ib_addr                             true        true

    ib_iser                             true        true

    ~ #

    6. Make sure that the drivers will be loaded after reboot

    ~ # esxcli system module set -e true -m mlx4_en

    ~ # esxcli system module set -e true -m mlx4_core

    ~ # esxcli system module set -e true -m mlx4_ib

     

    7. Verify VSwitch uplink connectivity.

    Get the vSwitch standard list patameters.

    Make sure that the uplink nic is mapped to the Mellanox adapter card.

    In this case vmnic0 is mapped to Mellanox ConnectX-3 adapter card, while vmnic1 is mapped to the onboard 1G port.

     

    ~ # esxcli network vswitch standard list

    vSwitch0

       Name: vSwitch0

       Class: etherswitch

       Num Ports: 3072

       Used Ports: 4

       Configured Ports: 128

       MTU: 1500

       CDP Status: listen

       Beacon Enabled: false

       Beacon Interval: 1

       Beacon Threshold: 3

       Beacon Required By:

       Uplinks: vmnic1

       Portgroups: VM Network, Management Network

     

    ~ # esxcli network nic list

    Name          PCI Device     Driver   Link  Speed  Duplex  MAC Address         MTU  Description

    ------------  -------------  -------  ----  -----  ------  -----------------  ----  ---------------------------------------------------

    vmnic0        0000:002:00.0  mlx4_en  Up    40000  Full    00:02:c9:a0:9a:10  1500  Mellanox Technologies MT27500 Family [ConnectX-3]

    vmnic1        0000:005:00.0  e1000e   Up     1000  Full    00:25:90:4f:ea:fa  1500  Intel Corporation 82574L Gigabit Network Connection

    vmnic1000002  0000:002:00.0  mlx4_en  Down      0  Half    00:02:c9:a0:9a:11  1500  Mellanox Technologies MT27500 Family [ConnectX-3]

    vmnic2        0000:006:00.0  e1000e   Down      0  Half    00:25:90:4f:ea:fb  1500  Intel Corporation 82574L Gigabit Network Connection

    ~ #

     

    Add Mellanox uplink port.

    Verify that vmnic0 is one of the uplink port list.

     

    ~ # esxcli network vswitch standard uplink add -u vmnic0 -v vSwitch0

    ~ # esxcli network vswitch standard list

    vSwitch0

       Name: vSwitch0

       Class: etherswitch

       Num Ports: 3072

       Used Ports: 6

       Configured Ports: 128

       MTU: 1500

       CDP Status: listen

       Beacon Enabled: false

       Beacon Interval: 1

       Beacon Threshold: 3

       Beacon Required By:

       Uplinks: vmnic0, vmnic1

       Portgroups: VM Network, Management Network

    ~ #

     

    8. Use the vSphere client to login to the host, and check the configuration tab, you should see similar vSwitch diagram, with two uplink ports.

     

    1.PNG

    2.PNG

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

     

    9. Another option is to have two vSwitches, one for each adapter port. For example, one for the on-board 1GbE port and one for the ConnectX-3 40GbE port.

    It can be done via the GUI easily:

    In this example, vSwitch0 is using the management port uplink. while vSwitch1 is using the 40GbE ConnectX-3 port as uplink.

    In addition, VMKernel was configured on vSwitch1.

     

     

    3.PNG

     

     

    Troubleshooting

     

    1. To restart the driver run:

    ~ # /opt/mellanox/bin/openibd.sh restart

    /etc/init.d/sfcbd-watchdog stop

    vmkload_mod -u mlx4_ib

    Module mlx4_ib successfully unloaded

    vmkload_mod -u ib_iser

    Module ib_iser successfully unloaded

    vmkload_mod -u rdma_cm

    Module rdma_cm successfully unloaded

    vmkload_mod -u ib_addr

    Module ib_addr successfully unloaded

    vmkload_mod -u ib_cm

    Module ib_cm successfully unloaded

    vmkload_mod -u ib_umad

    Module ib_umad successfully unloaded

    vmkload_mod -u ib_sa

    Module ib_sa successfully unloaded

    vmkload_mod -u ib_mad

    Module ib_mad successfully unloaded

    vmkload_mod -u ib_core

    Module ib_core successfully unloaded

    vmkload_mod -u mlx4_en

    Module mlx4_en successfully unloaded

    vmkload_mod -u mlx4_core

    Module mlx4_core successfully unloaded

    vmkload_mod mlx4_core

    Module mlx4_core loaded successfully

    vmkload_mod mlx4_en

    Module mlx4_en loaded successfully

    vmkload_mod ib_core

    Module ib_core loaded successfully

    vmkload_mod ib_mad

    Module ib_mad loaded successfully

    vmkload_mod ib_sa

    Module ib_sa loaded successfully

    vmkload_mod ib_umad

    Module ib_umad loaded successfully

    vmkload_mod ib_cm

    Module ib_cm loaded successfully

    vmkload_mod ib_addr

    Module ib_addr loaded successfully

    vmkload_mod rdma_cm

    Module rdma_cm loaded successfully

    vmkload_mod ib_iser

    Module ib_iser loaded successfully

    vmkload_mod mlx4_ib

    Module mlx4_ib loaded successfully

    /etc/init.d/sfcbd-watchdog start

    ~ #