PeerDirect Programming: Create Context with Peer Attributes

Version 4

    New experimental verb ibv_exp_open_device is added to OFED 4.1 (ConnectX-4 and ConnectX-5 adapters) ib_core module.

    The following verb introduces the ability to open a device and create a context while giving PCI peer attributes (in ibv_exp_open_device_attr) such as name and ID.

    This post is technical and addresses developers.

     

    References

     

    Overview

    The following struct has been added as an experimental struct:

    struct ibv_context *ibv_exp_open_device(struct ibv_device *device, struct ibv_exp_open_device_attr *attr);

     

    By using this struct, the driver opens the device and creates a context with the specified attributes.

     

    The new experimental verb supports the following attribute: IBV_EXP_SET_ATTR_PEER_INFO

    PeerDirect users with non-unified virtual addresses should use this attribute to create a device context and attach it to peers.

    By using this attribute, the driver will be able to separate between host and peer addresses and handle its translation accordingly.

     

    Usage

    Enable this feature by passing ibv_exp_open_device_attr struct to the new experimental verbs.

    struct ibv_exp_open_device_attr {

                   uint32_t                  comp_mask;   /* Identifies valid fields. Use enum ibv_exp_set_context_attr */

                   struct exp_peer_info      peer_info;   /* Provides peer information - name , id */

           };

     

    exp_peer_info

    struct exp_peer_info {

              uint32_t               peer_id;

              char                   * peer_name;

    };

     

    Currently, the attribute compatibility is as follows:

    enum ibv_exp_set_context_attr{

            IBV_EXP_SET_ATTR_PEER_INFO              = 1 << 0, /* Set if attr->peer_info is valid */

            IBV_EXP_SET_CONTEXT_ATTR_RESERVED       = 1 << 1,

    };

    The enum might be extended in the future.

     

     

    By setting the IBV_EXP_SET_ATTR_PEER_INFO bit, if the input peer ID and peer name are correct, the newly created context will be assigned to the peer.

    To use the new verb with peer capability, two new peer callbacks should be implemented in io_peer_mem:

         1. void* (*get_context_private_data)(u64 peer_id)

             Receive peer_id and return peer context

     

         2.  void (*put_context_private_data)(void *context)

             Free peer context allocated by get_context_private_data

     

    Validation

    io_peer_mem module with the new callbacks should be implemented.