7 Replies Latest reply on Feb 18, 2014 9:16 AM by chuckleb

    40Gb Ethernet in VMWare ESX 5.5?

      I'm playing with some ConenctX-3 cards and seem to be able to connect two machines with 10Gb in ethernet mode just fine, but I saw a press release that said you can get 40Gbps out of these. Can anyone tell me how or tell me how to confirm what speeds I am getting? I currently see it as 10000 Full for speeds. Firmware has been updated to latest. This is using the drivers that came with 5.5.


      net-mlx4-core                  VMware    VMwareCertified   2014-01-26 

      net-mlx4-en                    VMware    VMwareCertified   2014-01-26 


      Press release talking about 40Gbps.


        • Re: 40Gb Ethernet in VMWare ESX 5.5?

          What is the card's model number?

          There are 10G only and 10/40G types.

            • Re: 40Gb Ethernet in VMWare ESX 5.5?

              The cards on both ends are:


              [root@basement mnt]# ibv_devinfo

              hca_id: mlx4_0

                      transport:                      InfiniBand (0)

                      fw_ver:                         2.30.8000

                      node_guid:                      0002:c903:009f:5a20

                      sys_image_guid:                 0002:c903:009f:5a23

                      vendor_id:                      0x02c9

                      vendor_part_id:                 4099

                      hw_ver:                         0x0

                      board_id:                       MT_1090110028

                      phys_port_cnt:                  1

                              port:   1

                                      state:                  PORT_DOWN (1)

                                      max_mtu:                4096 (5)

                                      active_mtu:             4096 (5)

                                      sm_lid:                 0

                                      port_lid:               0

                                      port_lmc:               0x00

                                      link_layer:             InfiniBand

            • Re: 40Gb Ethernet in VMWare ESX 5.5?


              Could you please run the following and provide the output:

              #esxcli network nic list


                • Re: 40Gb Ethernet in VMWare ESX 5.5?

                  ~ # esxcli network nic list

                  Name          PCI Device     Driver   Link  Speed  Duplex  MAC Address         MTU  Description                                      

                  ------------  -------------  -------  ----  -----  ------  -----------------  ----  --------------------------------------------------

                  vmnic0        0000:001:00.0  igb      Up     1000  Full    00:a0:d1:ec:a7:dc  1500  Intel Corporation 82576 Gigabit Network Connection

                  vmnic1        0000:001:00.1  igb      Down      0  Half    00:a0:d1:ec:a7:dd  1500  Intel Corporation 82576 Gigabit Network Connection

                  vmnic1000202  0000:004:00.0  mlx4_en  Down      0  Half    00:02:c9:9f:55:b1  1500  Mellanox Technologies MT27500 Family [ConnectX-3]

                  vmnic2        0000:004:00.0  mlx4_en  Up    10000  Full    00:02:c9:9f:55:b0  9000  Mellanox Technologies MT27500 Family [ConnectX-3]


                  As an aside note, are the Mellanox EN drivers bundled in the ESX 5.5 install newer/better than the download links for 1.6.1?

                • Re: 40Gb Ethernet in VMWare ESX 5.5?



                  The card you have displayed MT_1090110028 is capable of 40gbps InfiniBand and 10gbps Ethernet.  it is not capable of 40gbps Ethernet.  Though we do have cards that are capable of 40gbps Ethernet.


                  ConnectX-3 VPI adapter card; dual-port QSFP; FDR10 IB (40Gb/s) and 10GigE; PCIe3.0x8 8GT/s; RoHS R6

                  PCI DevID: 4099

                    • Re: 40Gb Ethernet in VMWare ESX 5.5?

                      Thanks for pointing that out! I hadn't realized that there were so many variants of the card. So if I were to get a different card that was 40GbE capable, would ESX detect it correctly using the stock drivers? No other configuration needed?




                        • Re: 40Gb Ethernet in VMWare ESX 5.5?

                          Closing the loop. I purchased a pair of the 40GB VPI with 40GbE on it and swapped cards. ESXi and CentOS immediately saw them as 40 Gbps connections. No updating needed.


                          Followup question.


                          The mlx4_en driver in ESX 5.5 stock reports as:

                          ~ # ethtool -i vmnic2

                          driver: mlx4_en

                          version: (Dec-03-2012)

                          firmware-version: 2.30.8000

                          bus-info: 0000:04:00.0



                          Is this newer than the one provided on the Mellanox and Vmware links

                          (Mellanox Technologies: VMware: Firmware - Driver Compatibility Matrix0? It appears to according to version number and date.


                          If so, should I continue to use or should I downgrade to 1.6.1-2? Can I enable all the features and whatnot with the stock 1.9.7 driver?