1 Reply Latest reply on Jun 13, 2018 5:07 AM by vlad

    ESXI 6.7 nmlx drivers

    time81

      Hi,

       

      im wondering whats the difference between my hosts and drivers. Since i have massive speed problems when using storage vmotion and im trying to find the best driver/software version but i cant figure out the source of the problem

       

      3 HP DL380 G9 with Mellanox Technologies MT27700 Family [ConnectX-4]

       

      Remove old net-mlx4 when using 5 ? Do i need both ?    Why has the current 6.7 stock driver version 1.9.7 when the latest mlx driver package from homepage has 4.17 ?

       

      Is the VMWare 6.7 known issue the problem ?:

       

      The Mellanox ConnectX-4/ConnectX-5 native ESXi driver might exhibit performance degradation when its Default Queue Receive Side Scaling (DRSS) feature is turned on

      Receive Side Scaling (RSS) technology distributes incoming network traffic across several hardware-based receive queues, allowing inbound traffic to be processed by multiple CPUs. In Default Queue Receive Side Scaling (DRSS) mode, the entire device is in RSS mode. The driver presents a single logical queue to OS and is backed by several hardware queues.

      The native nmlx5_core driver for the Mellanox ConnectX-4 and ConnectX-5 adapter cards enables the DRSS functionality by default. While DRSS helps to improve performance for many workloads, it could lead to possible performance degradation with certain multi-VM and multi-vCPU workloads.

      Workaround: If significant performance degradation is observed, you can disable the DRSS functionality.

      1. Run the esxcli system module parameters set -m nmlx5_core -p DRSS=0 RSS=0 command.
      2. Reboot the host.

       

       

       

      Host1 (working fast)

      NetAppNasPlugin                1.1.0-0                               NetApp           VMwareAccepted    2016-12-06

      net-mlx4-core                  1.9.7.0-1vmw.670.0.0.8169922          VMW              VMwareCertified   2018-04-27

      net-mlx4-en                    1.9.7.0-1vmw.670.0.0.8169922          VMW              VMwareCertified   2018-04-27

       

      Host2 SLOW

       

      NetAppNasPlugin                1.1.0-0                               NetApp           VMwareAccepted    2016-12-05

      nmlx4-core                     3.17.9.12-1vmw.670.0.0.8169922        VMW              VMwareCertified   2018-04-27

      nmlx4-en                       3.17.9.12-1vmw.670.0.0.8169922        VMW              VMwareCertified   2018-04-27

      nmlx4-rdma                     3.17.9.12-1vmw.670.0.0.8169922        VMW              VMwareCertified   2018-04-27

      nmlx5-core                     4.17.9.12-1vmw.670.0.0.8169922        VMW              VMwareCertified   2018-04-27

      nmlx5-rdma                     4.17.9.12-1vmw.670.0.0.8169922        VMW              VMwareCertified   2018-04-27

       

      Host 3 (Partly slow)

       

      nmst                           4.7.0.42-1OEM.650.0.0.4598673         MEL              PartnerSupported  2017-08-09

      mft                            4.7.0.42-0                            Mellanox         PartnerSupported  2017-08-09

      NetAppNasPlugin                1.1.2-3                               NetApp           VMwareAccepted    2017-08-18

      nmlx4-core                     3.17.9.12-1vmw.670.0.0.8169922        VMW              VMwareCertified   2018-04-27

      nmlx4-en                       3.17.9.12-1vmw.670.0.0.8169922        VMW              VMwareCertified   2018-04-27

      nmlx4-rdma                     3.17.9.12-1vmw.670.0.0.8169922        VMW              VMwareCertified   2018-04-27

      nmlx5-core                     4.17.9.12-1vmw.670.0.0.8169922        VMW              VMwareCertified   2018-04-27

      nmlx5-rdma                     4.17.9.12-1vmw.670.0.0.8169922        VMW              VMwareCertified   2018-04-27

       

      thanks for the help

        • Re: ESXI 6.7 nmlx drivers
          vlad

          Hi Tim.

           

          1. ESXi 6.7 has Mellanox driver as inbox. ( in your case : nmlx5-core + nmlx5-rdma modules for ConnectX-4, ConnectX-5 family)
          2. nmlx4-* modules are of native driver for ConnectX-3 NIC family (in your case can be removed)

          3. net-mlx4-* modules are of vmkinux driver (old API) for ConnectX-3 NIC family  (in your case can be removed)

          4. In general, it's always advised to install latest native driver from (when available) : http://www.mellanox.com/page/products_dyn?product_family=29

          5. About performance there is too little information about configuration and actual bandwidth, kindly elaborate.