1 Reply Latest reply on Nov 17, 2015 9:16 AM by ophirmaor

    vSphere ESXi VLANs

    adamvdc

      Hi,

       

      I am trying to set up an Infiniband network to support 10Gbps IPoIB for my vSphere 6 environment.

       

      I have 2 ESXi 6.0.0 hosts with the OFED 2.3.3.1 installed each with a Voltaire 500 EX-D dual port HCA.

       

      These are connected to a SilverStorm 9024-CU24-ST2-DDR switch

       

      I can see the adapters and have created my vSwitch with port groups.

       

      I can ping between the hosts when connected to a port group with no VLAN set

       

      I want to configure VLANs/partitions, I have created several port groups on my vSwitch with VLAN IDs and created pkeys on the subnet manager within the 9025 switch but I am unable to get any traffic to work over these.

       

      I have read about using 0x8000 + VLAN ID so have the following set up:

       

      -> smPKeys

      Index:    0 PKey: 0xffff   Desc: Default PKey

      Index:    1 PKey: 0x8010   Desc: 10

      Index:    2 PKey: 0x8011   Desc: 11

      Index:    3 PKey: 0x8009   Desc: 9

       

      I have also tried using 0x0000 + VLAN ID

       

      In my vmkernel.log I am seeing things like:

       

      2015-11-15T19:43:59.734Z cpu2:33083)<4>vmnic1:ipoib_mcast_join_complete:642: multicast join failed for ff:12:40:1b:80:10:00:00:00:00:00:00:ff:ff:ff:ff, status -22

      2015-11-15T19:43:59.734Z cpu1:33075)<4>vmnic1000102:ipoib_mcast_join_complete:642: multicast join failed for ff:12:40:1b:80:10:00:00:00:00:00:00:ff:ff:ff:ff, status -22

       

      2015-11-15T19:34:31.412Z cpu3:33080)<3>vmnic1000102:ipoib_vmk_def_init:1516: failed to create Multicast QP for pkey = 9

      2015-11-15T19:34:31.417Z cpu3:33080)<3>vmnic1:ipoib_mcast_restart_task:1017: Unable to find Broadcast QP for pkey = 9

      2015-11-15T19:34:31.417Z cpu3:33080)<3>vmnic1:ipoib_mcast_restart_task:1023: Unable to find Multicast QP for pkey = 9

      2015-11-15T19:34:31.417Z cpu3:33080)<3>vmnic1000102:ipoib_mcast_restart_task:1017: Unable to find Broadcast QP for pkey = 9

      2015-11-15T19:34:31.417Z cpu3:33080)<3>vmnic1000102:ipoib_mcast_restart_task:1023: Unable to find Multicast QP for pkey = 9

       

      I can see the pkeys I have created appear on the esxi hosts:

       

      [root@esx21:/var/log] grep -v "0000" /proc/sysfs/devices/virtual/infiniband/mlx4_0/ports/2/pkeys/*

      /proc/sysfs/devices/virtual/infiniband/mlx4_0/ports/2/pkeys/0:0xffff

      /proc/sysfs/devices/virtual/infiniband/mlx4_0/ports/2/pkeys/1:0x8010

      /proc/sysfs/devices/virtual/infiniband/mlx4_0/ports/2/pkeys/2:0x8011

      /proc/sysfs/devices/virtual/infiniband/mlx4_0/ports/2/pkeys/3:0x8009

       

      Have I missed a step somewhere or am I doing something wrong?

       

      Any help would be appreciated.

        • Re: vSphere ESXi VLANs
          ophirmaor

          I see here few issues:

          1. Version 2.3.3.1 explicitly doesn’t support ESX 6.0, it supports ESX 5.5 only.

          2. IPoIB multicast on 2.3.3.1 (assuming you will move to ESX 5.5) use DMFS (flow steering) by default.

          3. Need to verify that you use the latest FW mentioned in the Release Notes (last GA ConnectX-3 FW).

           

          I hope that it helps.

           

          Ophir.