HowTo Use InfiniBand PKEY Membership Types in Virtualization Environment

Version 5

    This post discusses the membership types of InfiniBand PKEYs and show a usage example in virtualization environment.

    The post is basic and meant for IT managers/architects that have some experience with InfiniBand networks.

     

    References

     

    Overview

    Each Partition Key (PKEY) is associated with multiple members (endpoints). There are two types of membership: limited or full.

    Limited members cannot accept information from other Limited members, but communication is allowed between every other combination of membership types.

    The membership type is added as the most-significant bit to the PKEY number. For example, the default PKEY will either have a value of 0x7FFF (limited) or 0xFFFF (full).

     

    There is an option to assign both full and limited PKEYs (allow both pkeys). This option will create both types of PKEYs on the member side. This is useful  in virtualized environment: The hypervisor (PF) will be able to assign each VM (VF) either limited or full PKEY. For example, VMs that are considered as "clients" will get limited PKEYs, and VM that is considered as "server" will get full PKEYs. Now both clients can communicate with the server, but not with each other. This will enable isolation of tenants.

     

    Configuration

    There are two configuration procedure to allow both PKEYs, depends on the location of the SM, either on a host or on a switch.

     

    Configure partition membership type on a Switch SM

     

    In case the Switch SM is used, follow this procedure:

     

    1. Allow "both" PKEYs

    switch (config) # ib sm allow-both-pkeys

     

    2. Restart the SM

    switch (config) # no ib sm

    switch (config) # ib sm

     

    Verify:

    switch (config) # show ib sm allow-both-pkeys

    Enabled

     

    3. Change default partition to "both" :

    switch (config) # ib partition Default defmember both

    Warning: you are about to make changes to the Default partition.

    Please notice that any such change may break connectivity to some nodes in the fabric or to the whole fabric.

    Type 'yes' to continue: yes

     

    switch (config) # ib partition Default member all type both

    Warning: you are about to make changes to the Default partition.

    Please notice that any such change may break connectivity to some nodes in the fabric or to the whole fabric.

    Type 'yes' to continue: yes

     

    4. Create a non-default partition

    switch (config) # ib partition par1 pkey 0x1

    switch (config) # ib partition par1 defmember both

    switch (config) # ib partition par1 member all type both

     

    Verify configuration:

     

    switch (config) # show ib partition

      Default

        PKey      = 0x7FFF

        defmember = both

        ipoib     = yes

      members

        GUID='ALL' member='both'

      par1

        PKey      = 0x0001

        defmember = both

        ipoib     = no

      members

        GUID='ALL' member='both'

    Configure partition membership type on a SM installed on a host

    In case a host SM is used follow this procedure:

     

    1. Update /etc/opensm/partitions.conf in the following manner:

    Default=0xffff,ipoib, defmember=both : ALL=both;

    Part1=0x1,ipoib, defmember=both : ALL=both;

     

    2. Run opensmd with the flag -W or --allow_both_pkeys

    # opensm --allow_both_pkeys

     

    Alternatively, update the following line in /etc/opensm/opensm.conf

    allow_both_pkeys TRUE

     

    and restart the OpenSM deamon :

    # service opensmd restart

     

    3. Now you can connect to any of the hosts to verify the PKEY configuration:

    # cat /sys/class/infiniband/mlx4_0/ports/1/pkeys/* |grep -v 0000

    0xffff

    0x8001

    0x7fff

    0x0001

     

    Note: Each PKEY appears twice (e.g. 0x8001, 0x0001), once as limited (MSB=0) and once as full (MSB=1)

     

     

    Virtualization PKEY Mapping

    Once the both option is configured, you can map limited PKEYs to specific VMs (e.g. clients) while full PKEYs to other VMs (e.g. server)

     

    1. Assign public PKEYs (full) on the hypervisor for VM1 (e.g. server) and private PKEYs (limited) for VM2 (e.g. client)

    # cd /sys/class/infiniband/mlx4_0/iov/

    # echo 0 > 0000:02:00.1/ports/1/pkey_idx/0      --> This will assign 0xffff to VM1-Server

    # echo 1 > 0000:02:00.1/ports/1/pkey_idx/1      --> This will assign 0x8001 to VM1-Server

    # echo 2 > 0000:02:00.2/ports/1/pkey_idx/0      --> This will assign 0x7fff to VM2-Client

    # echo 3 > 0000:02:00.2/ports/1/pkey_idx/1      --> This will assign 0x0001 to VM2-Client

     

    2. Verify on the VM1-Server:

    [VM1-Server]# cat /sys/class/infiniband/mlx4_0/ports/1/pkeys/* |grep -v 0000

    0xffff

    0x8001

     

    3. Verify on the VM2-Client:

    [VM2-Client]# cat /sys/class/infiniband/mlx4_0/ports/1/pkeys/* |grep -v 0000

    0x7fff

    0x0001