7 Replies Latest reply on Jun 21, 2014 2:03 AM by adrianidsys

    How to set IPoIB interface MTU in VmWare 5.5




      I do have a problem setting the IPoIB interface MTU in VmWare 5.5 using the 1.8.2 drivers.


      The adaptors i use are MCX354A-FCBT (dual-port IB/VPI adaptors - ConnectX3). They have been flashed with the latest firmware 2.30.8000 recently. The same issue applied when using previous firmware.


      I have set the mtu_4k parameter to 1 for the mlx4_core module and restarted several time.


      The issue I'm having is when setting the 4092 MTU value for the vSwitch who has the vmnic_ib0 adaptor as uplink, i'm getting the:

      "Unable to set MTU to 4092 the following uplinks refused the MTU setting: vmnic_ib0" message.


      Could you please explain if there are additional steps required for this? Am I missing something?


      Thanks in advance..

        • Re: How to set IPoIB interface MTU in VmWare 5.5


          Your procedure was correct!

          And there is plus 1 step!

          That's a IB Partition configuration.

          If you complete partition configuration restart SM then you can configura vSwitch MTU to 4092 bytes.

            • Re: How to set IPoIB interface MTU in VmWare 5.5

              Hi there,

              Changes in the MTU size for an IPoIB pkey can be tricky. Its always possible to decrease, but increasing might not.

              The first device connected will set the MTU size for the broadcast domain. i.e. subsequent connection of IPoIB interfaces will not be allowed to set an MTU size larger than the previous one.

              So my suggestion is:

              -disconnect all IPoIB interfaces attached to the partition (pkey interfaces)

              - configure Subentmanager with the requested MTU size (like MTU=5 in partition.conf)

              -restart SM

              -make sure that the first interface added to the partition  will use mtu size 4092

              -- example: in linux I would do as follows

                 echo 0x0001 > /sys/class/net/ib0/create_child

                 echo 4092 > /sys/class/net/ib0.8001/mtu

                 ifconfig ib0.8001 mtu 4092 up

                 and then check that the mtu really IS 4092 before adding the interfaces on the second node.


              ref: RFC4391 Transmission of IP over InfiniBand (IPoIB)

              5. Setting Up an IPoIB Link

              The join operation is successful only if the Subnet Manager (SM)

                determines that the joining node can support the MTU registered with

                the broadcast group [RFC4392] ensuring support for a common link MTU.

            • Re: How to set IPoIB interface MTU in VmWare 5.5

              Sorry that I am digging up an answered question, but I think I have the same problem with my setup.


              I am running Vmware ESXi 5.1 with DDR IB interfaces, connected to a HP 4x  DDR Switch, I have installed SM on the ESXi host (opensm) and I cant manage to set up the MTU = 4k

              That's the error, vmware is giving me :

              Call "HostNetworkSystem.UpdateVirtualSwitch" for object "networkSystem" on ESXi "" failed.

              Operation failed, diagnostics report: Unable to set MTU to 4092 the following uplinks refused the MTU setting: vmnic_ib0


              The mtu_4k is set to 1, the partitions.conf file is as follows : Default=0x7fff,ipoib,mtu=5:ALL=full;

              The weird part is that I was able to configure it on one of the hosts, but can't do it on the others.

              Any ideas on how to make the other hosts use mtu = 4092 ?

              Thanks a lot in advance!

              P.S I am using the MLNX-OFED-ESX- version of the driver.