Understanding Linux debug-fs Flow Steering rules

Version 1

    This post discusses the location of the flow steering rules located in /sys/kernel/debug.

    In the example below we will create few flow steering rules and show the rules in the file system. some applications may wish to monitor those rules.

     

    There are few options to get the current flow steering tables.

    1. Via ethtool. The problem with this method is that it shows only flows that were configured via ethtool.

    2. Via mlx_fs_dump, this script shows the configuration in the firmware.

    3. Via debug fs (/sys/kernel/debug) - shown below.

     

    This post is meant for developers or advance users.

     

    References

     

    Configuration

     

    1. Install the latest MLNX_OFED.

     

    2. Create a flow steering rule.

    # ethtool -u eth1

    16 RX rings available

    Total 0 rules

     

    # ethtool -U eth1 flow-type udp4 dst-port 319 action 0 loc 1

    # ethtool -U eth1 flow-type udp4 dst-port 320 action 0 loc 2

     

    # ethtool -u eth1

    16 RX rings available

    Total 2 rules

     

     

    Filter: 1

      Rule Type: UDP over IPv4

      Src IP addr: 0.0.0.0 mask: 255.255.255.255

      Dest IP addr: 0.0.0.0 mask: 255.255.255.255

      TOS: 0x0 mask: 0xff

      Src port: 0 mask: 0xffff

      Dest port: 319 mask: 0x0

      Action: Direct to queue 0

     

    Filter: 2

      Rule Type: UDP over IPv4

      Src IP addr: 0.0.0.0 mask: 255.255.255.255

      Dest IP addr: 0.0.0.0 mask: 255.255.255.255

      TOS: 0x0 mask: 0xff

      Src port: 0 mask: 0xffff

      Dest port: 320 mask: 0x0

      Action: Direct to queue 0

     

    3. Get the tree of the following location.

    # tree /sys/kernel/debug/mlx5/0000\:02\:00.0/steering/root_ns/priorities/ethtool/namespaces/ethtool_ns/

     

    /sys/kernel/debug/mlx5/0000:02:00.0/steering/root_ns/priorities/ethtool/namespaces/ethtool_ns/

    |-- priorities

    |   |-- prio_0

    |   |   |-- flow_tables

    |   |   |-- namespaces

    |   |   |-- priority

    |   |   |-- refcount

    |   |   `-- type

    |   |-- prio_1

    |   |   |-- flow_tables

    |   |   |-- namespaces

    |   |   |-- priority

    |   |   |-- refcount

    |   |   `-- type

    |   |-- prio_10

    |   |   |-- flow_tables

    |   |   |-- namespaces

    |   |   |-- priority

    |   |   |-- refcount

    |   |   `-- type

    |   |-- prio_2

    |   |   |-- flow_tables

    |   |   |-- namespaces

    |   |   |-- priority

    |   |   |-- refcount

    |   |   `-- type

    |   |-- prio_3

    |   |   |-- flow_tables

    |   |   |-- namespaces

    |   |   |-- priority

    |   |   |-- refcount

    |   |   `-- type

    |   |-- prio_4

    |   |   |-- flow_tables

    |   |   |-- namespaces

    |   |   |-- priority

    |   |   |-- refcount

    |   |   `-- type

    |   |-- prio_5

    |   |   |-- flow_tables

    |   |   |   `-- flow_table_9

    |   |   |       |-- groups

    |   |   |       |   `-- group_21

    |   |   |       |       |-- entries

    |   |   |       |       |   |-- fte0

    |   |   |       |       |   |   |-- action

    |   |   |       |       |   |   |-- destinations

    |   |   |       |       |   |   |   `-- dest_tir_157

    |   |   |       |       |   |   |       |-- dest_type

    |   |   |       |       |   |   |       |-- refcount

    |   |   |       |       |   |   |       |-- tir

    |   |   |       |       |   |   |       `-- type

    |   |   |       |       |   |   |-- dests_size

    |   |   |       |       |   |   |-- flow_tag

    |   |   |       |       |   |   |-- index

    |   |   |       |       |   |   |-- refcount

    |   |   |       |       |   |   |-- type

    |   |   |       |       |   |   `-- value

    |   |   |       |       |   |       |-- inner_headers

    |   |   |       |       |   |       `-- outer_headers

    |   |   |       |       |   |           |-- ethertype

    |   |   |       |       |   |           |-- ip_protocol

    |   |   |       |       |   |           `-- udp_dport

    |   |   |       |       |  `-- fte1

    |   |   |       |       |       |-- action

    |   |   |       |       |       |-- destinations

    |   |   |       |       |       |   `-- dest_tir_157

    |   |   |       |       |       |       |-- dest_type

    |   |   |       |       |       |       |-- refcount

    |   |   |       |       |       |       |-- tir

    |   |   |       |       |       |       `-- type

    |   |   |       |       |       |-- dests_size

    |   |   |       |       |       |-- flow_tag

    |   |   |       |       |       |-- index

    |   |   |       |       |       |-- refcount

    |   |   |       |       |       |-- type

    |   |   |       |       |       `-- value

    |   |   |       |       |           |-- inner_headers

    |   |   |       |       |           `-- outer_headers

    |   |   |       |       |               |-- ethertype

    |   |   |       |       |               |-- ip_protocol

    |   |   |       |       |               `-- udp_dport

    |   |   |       |       |-- id

    |   |   |       |       |-- mask

    |   |   |       |       |   |-- inner_headers

    |   |   |       |       |   |-- misc_params

    |   |   |       |       |   `-- outer_headers

    |   |   |       |       |       |-- ethertype

    |   |   |       |       |       |-- ip_protocol

    |   |   |       |       |       `-- udp_dport

    |   |   |       |       |-- match_criteria_enable

    |   |   |       |       |-- max_ftes

    |   |   |       |       |-- num_ftes

    |   |   |       |       |-- refcount

    |   |   |       |       |-- start_index

    |   |   |       |       `-- type

    |   |   |       |-- id

    |   |   |       |-- level

    |   |   |       |-- max_fte

    |   |   |       |-- refcount

    |   |   |       `-- type

    |   |   |-- namespaces

    |   |   |-- priority

    |   |   |-- refcount

    |   |   `-- type

    |   |-- prio_6

    |   |   |-- flow_tables

    |   |   |-- namespaces

    |   |   |-- priority

    |   |   |-- refcount

    |   |   `-- type

    |   |-- prio_7

    |   |   |-- flow_tables

    |   |   |-- namespaces

    |   |   |-- priority

    |   |   |-- refcount

    |   |   `-- type

    |   |-- prio_8

    |   |   |-- flow_tables

    |   |   |-- namespaces

    |   |   |-- priority

    |   |   |-- refcount

    |   |   `-- type

    |   `-- prio_9

    |       |-- flow_tables

    |       |-- namespaces

    |       |-- priority

    |       |-- refcount

    |       `-- type

    |-- refcount

    `-- type

     

    4. Ge the tree of specific flow:

    # tree /sys/kernel/debug/mlx5/0000\:02\:00.0/steering/root_ns/priorities/ethtool/namespaces/ethtool_ns/priorities/prio_5/flow_tables/flow_table_9/groups/group_21/mask/outer_headers/

    /sys/kernel/debug/mlx5/0000:02:00.0/steering/root_ns/priorities/ethtool/namespaces/ethtool_ns/priorities/prio_5/flow_tables/flow_table_9/groups/group_21/mask/outer_headers/

    |-- ethertype

    |-- ip_protocol

    `-- udp_dport

     

    5. Get specific value (e.g. udp port):

    # cat /sys/kernel/debug/mlx5/0000\:02\:00.0/steering/root_ns/priorities/ethtool/namespaces/ethtool_ns/priorities/prio_5/flow_tables/flow_table_9/groups/group_21/entries/fte1/value/outer_headers/udp_dport

    320