Raw Ethernet Programming: "Don't trap" Flow Steering - Code Example

Version 5

    This post shows how to steer a packet using the "don't trap" flow steering rule, when developing Raw Ethernet over verbs.

    The "don't trap" rule does not trap the received packets to the CPU, but instead, it allows them to match lower prioritized rules. The feature is mainly used for sniffing packets.


    This feature is available starting with upstream kernel 4.8. and above, and is planned for MLNX_OFED 4.0

    The feature is supported by ConnectX-4 and ConnectX-3 (with some limitations).




    In order to create the don't trap rule,  you must use the ibv_flow_attr struct and call the ibv_cmd_create_flow.

    The application appends several specs for each feature at the end of the struct, as follows:


    struct ibv_flow_attr {

            enum ibv_flow_attr_type type;

            uint16_t size;

            uint16_t priority;

            uint8_t num_of_specs;

            uint8_t port;

            uint32_t flags;

            /* Following are the optional layers according to the user request

             * struct ibv_flow_spec_xxx [L2]

             * struct ibv_flow_spec_yyy [L3/L4]


            uint64_t reserved; /* reserved for future growth (must be 0) */



    The flow steering creation is made through a container struct ibv_flow_attr, which includes specification control header, followed by the flow specification itself.



    1. Set the number of specs to add 1. For example, if no specs were specified before, change num_of_specs from 0 to 1.

    2. Set flags to IBV_FLOW_ATTR_FLAGS_DONT_TRAP.


    For example, use Raw Ethernet Programming: Basic Introduction - Code Example as a reference, and perform the following changes in the receiver side.

    /* 12. Register steering rule to intercept packet to DEST_MAC and place packet in ring pointed by ->qp, but allowing them to match lower prioritized rules  */

        struct raw_eth_flow_attr {

            struct ibv_flow_attr        attr;

        } __attribute__((packed)) flow_attr = {

            .attr = {

                .comp_mask  = 0,

                .type       = IBV_FLOW_ATTR_NORMAL,

                .size       = sizeof(flow_attr),

                .priority   = 0,

                .num_of_specs   = 1,

                .port       = PORT_NUM,

                .flags      = IBV_FLOW_ATTR_FLAGS_DONT_TRAP,



        /* 13. Create steering rule */

        struct ibv_flow *eth_flow;

        eth_flow = ibv_cmd_create_flow(qp, &flow_attr.attr);

        if (!eth_flow) {

            fprintf(stderr, "Couldn't attach steering flow\n");