How To Install/Upgrade Mellanox NATIVE ESXi Driver for VMware vSphere ESXi 6.5/6.7 host (CLI)

Version 21

    This is a basic post that shows how to install and test the Mellanox ConnectX-4/ConnectX-5 NATIVE ESXi Driver for VMware vSphere ESXi 6.5/6.7 on single host, and perform basic initial configuration steps to enable the driver by using ESXi cli interface.

     

    References

    Hardware and Software Requirements

    1. A server platform with an adapter card based on one of the following Mellanox Technologies’ HCA devices:

    2. Installer Privileges: The installation requires administrator privileges on the target machine.

    3. Device ID: For the latest list of device IDs, please visit Mellanox website.

    4. Supported NICs / Firmware: Recommended firmware versions you can find here.

        How To Update Firmware for ConnectX®-4/5 VPI PCI Express Adapter Cards (InfiniBand, Ethernet, VPI) on VMware ESXi 6.5/6.7.

     

     

    Driver Installation (CLI)

    1. Enable SSH Access to ESXi server.

    2. Log into ESXi vSphere Command-Line Interface with root permissions.

    3. Verify that the host is equipped with Mellanox adapter.

    ~ lspci | grep Mellanox

    0000:02:00.0 Network controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [vmnic2]

    0000:02:00.1 Network controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex] [vmnic3]

    Note: in this case, Mellanox card is using vmnic2 and vmnic3.

    4. Verify the driver's version was installed.

    ESXi 6.5

    ~ esxcli software vib list | grep nmlx
    nmlx5-core                     4.16.13.5-10EM-650.0.0-8277601        VMW       VMwareCertified   2018-04-24
    nmlx5-rdma                     4.16.13.5-10EM-650.0.0-8277601        VMW       VMwareCertified   2018-04-24

    ESXi 6.7

    ~ esxcli software vib list | grep nmlx

    nmlx4-core                     3.17.9.12-1vmw.670.0.0.8169922        VMW       VMwareCertified   2018-04-25

    nmlx4-en                       3.17.9.12-1vmw.670.0.0.8169922        VMW       VMwareCertified   2018-04-25

    nmlx4-rdma                     3.17.9.12-1vmw.670.0.0.8169922        VMW       VMwareCertified   2018-04-25

    nmlx5-core                     4.17.9.12-1vmw.670.0.0.8169922        VMW       VMwareCertified   2018-04-25

    nmlx5-rdma                     4.17.9.12-1vmw.670.0.0.8169922        VMW       VMwareCertified   2018-04-25

    5. Download a latest Mellanox native ESXi drivers from here.

    6.  Use SCP or any other file transfer method to copy the driver to the required ESXi host.

    7. Install the driver.

      ~ esxcli software vib install –d <path>/<bundle_file>

    ESXi 6.5

    ~ esxcli software vib install -d /tmp/MLNX-NATIVE-ESX-ConnectX-4-5_4.16.13.5-10EM-650.0.0.4598673.zip

    ESXi 6.7

    ~ esxcli software vib install -d /tmp/MLNX-NATIVE-ESX-ConnectX-4-5_4.17.13.8-10EM-670.0.0.8169922.zip

    8. Enter Maintenance Mode the ESXi host.

    9. Reboot the server.

    ~ reboot

    10. Verify the driver modules was installed and loaded successfully.

    esxcli software vib list | grep nmlx

    nmlx5-core                     4.17.13.8-1OEM.670.0.0.8169922        MEL       VMwareCertified   2018-06-07

    nmlx5-rdma                     4.17.13.8-1OEM.670.0.0.8169922        MEL       VMwareCertified   2018-06-07

    nmlx4-core                     3.17.9.12-1vmw.670.0.0.8169922        VMW       VMwareCertified   2018-06-06

    nmlx4-en                       3.17.9.12-1vmw.670.0.0.8169922        VMW       VMwareCertified   2018-06-06

    nmlx4-rdma                     3.17.9.12-1vmw.670.0.0.8169922        VMW       VMwareCertified   2018-06-06

    ~ esxcli system module list | grep nmlx5

    nmlx5_core                          true        true

    nmlx5_rdma                          true        true

    11. Check physical network interface status.

    ~ esxcli network nic list

    Name    PCI Device    Driver      Admin Status  Link Status  Speed   Duplex  MAC Address        MTU   Description

    ------  ------------  ----------  ------------  -----------  ------  ------  -----------------  ----  ----------------------------------------------------

    vmnic0  0000:81:00.0  igbn        Up            Up             1000  Full    0c:c4:7a:e3:5c:8c  1500  Intel Corporation I350 Gigabit Network Connection

    vmnic1  0000:81:00.1  igbn        Up            Down              0  Half    0c:c4:7a:e3:5c:8d  1500  Intel Corporation I350 Gigabit Network Connection

    vmnic2  0000:02:00.0  nmlx5_core  Up            Up           100000  Full    ec:0d:9a:8c:c7:96  1500  Mellanox Technologies MT28800 Family [ConnectX-5 Ex]

    vmnic3  0000:02:00.1  nmlx5_core  Up            Down              0  Half    ec:0d:9a:8c:c7:97  1500  Mellanox Technologies MT28800 Family [ConnectX-5 Ex]

    12. Exit Maintenance Mode the ESXi host.

     

    Done!