14 Replies Latest reply on Feb 8, 2014 6:26 AM by inbusiness

    vSphere 5.1 U1 IPoIB VLAN support

    inbusiness

      I want build a IPoIB VLAN virtual network via VST.

      I'm also made a partition and each vSwitch setting was completed.

      But just only VLAN 0, 21 works fine.

       

      vSphere 4.x was greate works!

      But vSphere 5.1 U1 wasn't...

       

      Do you have any solutions?

        • Re: vSphere 5.1 U1 IPoIB VLAN support
          ali

          Hi inbusiness,

          The way of VLAN configuration for IPoIB on vSphere 4.x/5.x didn't change.

          I suspect that you may have the wrong configuration on 5.x.

           

          Can you please provide more information.

          - Subnet Manager PKEY configuration file

          - vmnic configuration (esxcfg-vswitch -l on ESX, etc..)

            • Re: vSphere 5.1 U1 IPoIB VLAN support
              inbusiness

              * Here it is...

              ------------------------------------------------------------------------------------------------------------------

              ~ # esxcfg-vswitch -l

              Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks

              vSwitch0         128         6           128               1500    vmnic4,vmnic5

               

                PortGroup Name        VLAN ID  Used Ports  Uplinks

                Management Network    2934     1           vmnic4,vmnic5

               

              Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks

              vSwitch1         128         6           128               9000    vmnic0,vmnic1

               

                PortGroup Name        VLAN ID  Used Ports  Uplinks

                NFS                   0        1           vmnic0,vmnic1

               

              Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks

              vSwitch2         128         17          128               1500    vmnic2,vmnic3

               

                PortGroup Name        VLAN ID  Used Ports  Uplinks

                Simulation VM Network  2936     0           vmnic2,vmnic3

                VPN VM Network        2931     0           vmnic2,vmnic3

                Practice VM Network   2935     2           vmnic2,vmnic3

                External VM Network   0        2           vmnic2,vmnic3

                DHCP VM Network       2934     8           vmnic2,vmnic3

               

              Switch Name      Num Ports   Used Ports  Configured Ports  MTU     Uplinks

              vSwitch3         128         5           128               4092    vmnic_ib0

               

                PortGroup Name        VLAN ID  Used Ports  Uplinks

                iSCSI02 VM Network    36       0           vmnic_ib0

                iSCSI01 VM Network    35       0           vmnic_ib0

                Heartbeat VM Network  34       0           vmnic_ib0

                LM VM Network         33       0           vmnic_ib0

                CSV VM Network        32       0           vmnic_ib0

                FT                    17       1           vmnic_ib0

                vMotion               16       1           vmnic_ib0

               

              ~ #

              ------------------------------------------------------------------------------------------------------------------

              * Below is my 4036's partition.conf

              ------------------------------------------------------------------------------------------------------------------

              Default=  0xffff , ipoib , mtu=5 : ALL=full ;

              Partition10=0x10 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition11=0x11 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition12=0x12 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition13=0x13 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition14=0x14 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition15=0x15 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition16=0x16 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition17=0x17 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition18=0x18 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition19=0x19 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition20=0x20 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition21=0x21 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition22=0x22 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition23=0x23 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition24=0x24 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition25=0x25 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition26=0x26 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition27=0x27 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition28=0x28 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition29=0x29 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition30=0x30 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition31=0x31 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition32=0x32 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition33=0x33 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition34=0x34 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition35=0x35 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition36=0x36 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition37=0x37 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition38=0x38 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition39=0x39 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition40=0x40 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition41=0x41 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition42=0x42 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition43=0x43 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition44=0x44 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition45=0x45 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition46=0x46 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition47=0x47 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition48=0x48 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition49=0x49 , ipoib , mtu=5, defmember=full : ALL=full;

              ------------------------------------------------------------------------------------------------------------------

              * Below is result that response vmkping and ping each hosts

              (* checked partition in only works... )

              ------------------------------------------------------------------------------------------------------------------

              * Default=  0xffff , ipoib , mtu=5 : ALL=full ;

              Partition10=0x10 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition11=0x11 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition12=0x12 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition13=0x13 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition14=0x14 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition15=0x15 , ipoib , mtu=5, defmember=full : ALL=full;

              * Partition16=0x16 , ipoib , mtu=5, defmember=full : ALL=full; vMotion

              * Partition17=0x17 , ipoib , mtu=5, defmember=full : ALL=full; FT

              * Partition18=0x18 , ipoib , mtu=5, defmember=full : ALL=full; IPoIBSAN01

              * Partition19=0x19 , ipoib , mtu=5, defmember=full : ALL=full; IPoIBSAN02

              * Partition20=0x20 , ipoib , mtu=5, defmember=full : ALL=full;

              * Partition21=0x21 , ipoib , mtu=5, defmember=full : ALL=full;

              * Partition22=0x22 , ipoib , mtu=5, defmember=full : ALL=full;

              * Partition23=0x23 , ipoib , mtu=5, defmember=full : ALL=full;

              * Partition24=0x24 , ipoib , mtu=5, defmember=full : ALL=full;

              * Partition25=0x25 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition26=0x26 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition27=0x27 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition28=0x28 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition29=0x29 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition30=0x30 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition31=0x31 , ipoib , mtu=5, defmember=full : ALL=full;

              * Partition32=0x32 , ipoib , mtu=5, defmember=full : ALL=full; CSV VM

              * Partition33=0x33 , ipoib , mtu=5, defmember=full : ALL=full; LM VM

              * Partition34=0x34 , ipoib , mtu=5, defmember=full : ALL=full; Heartbeat VM

              * Partition35=0x35 , ipoib , mtu=5, defmember=full : ALL=full; iSCSI01 VM

              * Partition36=0x36 , ipoib , mtu=5, defmember=full : ALL=full; iSCSI02 VM

              * Partition37=0x37 , ipoib , mtu=5, defmember=full : ALL=full;

              * Partition38=0x38 , ipoib , mtu=5, defmember=full : ALL=full;

              * Partition39=0x39 , ipoib , mtu=5, defmember=full : ALL=full;

              * Partition40=0x40 , ipoib , mtu=5, defmember=full : ALL=full;

              * Partition41=0x41 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition42=0x42 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition43=0x43 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition44=0x44 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition45=0x45 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition46=0x46 , ipoib , mtu=5, defmember=full : ALL=full;

              Partition47=0x47 , ipoib , mtu=5, defmember=full : ALL=full;

              * Partition48=0x48 , ipoib , mtu=5, defmember=full : ALL=full;

              * Partition49=0x49 , ipoib , mtu=5, defmember=full : ALL=full;

              ------------------------------------------------------------------------------------------------------------------

              Default partition is always works.

              But next 6 partitions are not works and next 10 partitions are works and ...

               

              What's the my mistake?

            • Re: vSphere 5.1 U1 IPoIB VLAN support
              ingvar_j

              Well... when we configured our vSphere 5.1 just recently to run over IB, I found out that the decimal VLAN ID (1-4096) are converted to their hexadecimal equivalence when leaving the ESXi

              VLAN ID     Pkey

              1                   0x0001

              2                   0x0002

              9                   0x0009

              10                 0x000A    etc.

              and the ones I use

              230              0x00e6

              223              0x00df

               

              partition.conf on the subnetmanager host

              vmotion223=0xdf, ipoib, sl=0, defmember=full : 0x2590ffff07a3a9, 0x2590ffff07a3aa, 0x2590ffff07cb9d, 0x2590ffff07cb9e;

              pk0x0026=0xe6, ipoib, sl=0, defmember=full : SELF;

              NFS230=0xe6, ipoib, sl=0, defmember=full : 0x2c90300a7cff2, 0x2590ffff07a3a9, 0x2590ffff07a3aa, 0x2590ffff07cb9d, 0x2590ffff07cb9e;


              On the ESXi host, after configuring the VLAN 230 and 223 via the vCenter dSwitch

               

              # grep -v "0000"  /proc/infiniband/mlx4_0/ports/1/pkeys/*

              /proc/infiniband/mlx4_0/ports/1/pkeys/0:0xffff

              /proc/infiniband/mlx4_0/ports/1/pkeys/2:0x80e6

              /proc/infiniband/mlx4_0/ports/1/pkeys/3:0x80df

               

              So the VLAN ID 230 gets translated to PKEY 0x00E6, which in "member=FULL" mode = 0x80E6

               

              I hope this info will help. I could not find anything in the current documentation nor in previous versions. I did not test this on vSphere 4.1, so no idea how it was handled there.

              Regards, Ingvar

                • Re: vSphere 5.1 U1 IPoIB VLAN support
                  inbusiness

                  Thank you for your kind reply~!

                   

                  I'll try it!

                  • Re: vSphere 5.1 U1 IPoIB VLAN support
                    inbusiness

                    You are correct!

                     

                    I'm test 0x80e7.

                     

                    ~ # grep -v "0000"  /proc/infiniband/mlx4_0/ports/1/pkeys/*

                    /proc/infiniband/mlx4_0/ports/1/pkeys/0:0xffff

                    /proc/infiniband/mlx4_0/ports/1/pkeys/1:0x8001

                    /proc/infiniband/mlx4_0/ports/1/pkeys/2:0x8002

                    /proc/infiniband/mlx4_0/ports/1/pkeys/3:0x8003

                    /proc/infiniband/mlx4_0/ports/1/pkeys/4:0x8004

                    /proc/infiniband/mlx4_0/ports/1/pkeys/5:0x8005

                    /proc/infiniband/mlx4_0/ports/1/pkeys/6:0x8006

                    /proc/infiniband/mlx4_0/ports/1/pkeys/7:0x8007

                    /proc/infiniband/mlx4_0/ports/1/pkeys/8:0x8040

                    /proc/infiniband/mlx4_0/ports/1/pkeys/9:0x80e7

                     

                    9 means partition order by created in partitions.conf.

                    1 means HCA port number.

                    0x80e7 means real partition number.

                     

                    This case VLAN ID 9 wasn't worked, but 231 was working correctly...

                    (e7 is hexadecimal number of 231)

                     

                    Default partition number is 0x7ffff.

                    Therefore user partition number starts with 0x8.

                     

                    Thank you for your help.

                     

                    P.S

                    to Mellanox Community steps.

                    Can you describe this procedure in next vSphere OFED user guide?

                    It's a very important for your partner and admins.

                      • Re: vSphere 5.1 U1 IPoIB VLAN support
                        ingvar_j

                        Nice that You could use the information, and that You could verify also in your environment.

                        I agree that this information is essential, and should be in the documents.

                        Regards, Ingvar

                          • Re: vSphere 5.1 U1 IPoIB VLAN support
                            ingvar_j

                            By the way, I found the information in

                            https://git.ipxe.org/mirror/scst/.git/blob_plain?f=srpt/README

                            ""

                            SCSI RDMA Protocol (SRP) Target driver for Linux

                            =================================================

                            .....

                            InfiniBand Partitions

                            ---------------------

                             

                            Just like a VLAN allows to segment traffic on an Ethernet network partitions

                            allow to segment traffic on an InfiniBand network. Each InfiniBand partition

                            is identified by a partition key which is a 16-bit number. During fabric

                            initialization the subnet manager assigns one or more partition keys to

                            each InfiniBand port. For opensm partitions are defined in

                            /etc/opensm/partitions.conf. ib_srpt uses the partition with index 0. Which

                            partition key corresponds to index 0 can be found out by querying sysfs:

                             

                            $ head /sys/class/infiniband/*/ports/*/pkeys/0

                            ==> /sys/class/infiniband/mlx4_0/ports/1/pkeys/0 <==

                            0xffff

                             

                            ==> /sys/class/infiniband/mlx4_0/ports/2/pkeys/0 <==

                            0xffff

                            ----

                             

                            So in Linux it is stored under /sys/class/infiniband/ ,,,

                            and in ESXi under /proc/infiniband

                            The index file "0" holds the default 7fff pkey, and the following index entries 1,2,3 etc holds the created ones.

                            If the contetns of and index is 0x0000 it means that its unused. (i.e.  grep -v "0000 " will list the indexes holding configured values)