RoCE/RDMA Tools

Version 4

    This post provides a list of recommended Linux tools for configuring, monitoring and debugging RoCE traffic.

    The below information is applicable for Mellanox ConnectX-4 adapter cards and above, with the following SW: kernel version 4.11 and above, or  MLNX_OFED version 4.2 and above.

     

    Ethtool

    ethtool is a tool for querying and controlling network driver hardware settings.

    • All ethtool queries are applicable for RDMA traffic.

      • RDMA provides enhanced counters for counting RDMA traffic. A list of enhanced counters can be found here.

    • All ethtool configuration is applicable for RDMA traffic.

     

    ip link & ifconfig

    ip link and ifconfig are network device configuration and monitoring tools.

    • All ip link and ifconfig monitoring and configurations are applicable for RDMA.
    • Additional related monitoring and configuration tools:
      • gids - use show_gids tool (info)
      • ib info - use ibv_devinfo (man page)
      • rdma counters including congestion control - under /sys/class/infiniband/  and /sys/class/net locations (info)
      • congestion configuration - stored inside the device's non-volatile memory. If tuning is needed, it is recommended to seek Mellanox support. Explanations can be found here.

     

    tcpdump

    Dumps network traffic:

    Note: For RDMA, use ibdump (info)

     

    iperf3

    Tests network throughput.

    Note: For RDMA, use ib_send_bw (info) and ib_send_lat (info)

     

    netstat/ss

    Prints network connections, routing tables, interface statistics, masquerade connections and multicast memberships.

    Notes:

    • Most of the netstat information is applicable for RoCE
    • For open RDMA sockets use rdmatool (here)

     

    lldptool/dcbtool

    Manages the LLDP setting and status of lldpad (IEEE/CEE).

    • Handles lldptool-pfc (for lossless network) (here)