This post provides a list of recommended Linux tools for configuring, monitoring and debugging RoCE traffic.
The below information is applicable for Mellanox ConnectX-4 adapter cards and above, with the following SW: kernel version 4.11 and above, or MLNX_OFED version 4.2 and above.
ethtool is a tool for querying and controlling network driver hardware settings.
All ethtool queries are applicable for RDMA traffic.
RDMA provides enhanced counters for counting RDMA traffic. A list of enhanced counters can be found here.
- All ethtool configuration is applicable for RDMA traffic.
ip link & ifconfig
ip link and ifconfig are network device configuration and monitoring tools.
- All ip link and ifconfig monitoring and configurations are applicable for RDMA.
- Additional related monitoring and configuration tools:
- gids - use show_gids tool (info)
- ib info - use ibv_devinfo (man page)
- rdma counters including congestion control - under /sys/class/infiniband/ and /sys/class/net locations (info)
- congestion configuration - stored inside the device's non-volatile memory. If tuning is needed, it is recommended to seek Mellanox support. Explanations can be found here.
Dumps network traffic:
Note: For RDMA, use ibdump (info)
Tests network throughput.
Prints network connections, routing tables, interface statistics, masquerade connections and multicast memberships.
- Most of the netstat information is applicable for RoCE
- For open RDMA sockets use rdmatool (here)
Manages the LLDP setting and status of lldpad (IEEE/CEE).
- Handles lldptool-pfc (for lossless network) (here)