Understanding mlx5 Linux Counters and Status Parameters

Version 15

    This post discusses the Linux port counters and status parameters located under /sys/class/infiniband/  and /sys/class/net locations.

    The list of counters are aligned with MLNX_OFED 4.1.

     

    References

     

    Counter Updates

    MLNX_OFED 4.1 Updates

    1. MLNX_OFED 4.1 and kernel 4.12 (accepted kernel patch) adds 4 CNP/RoCE congestion counters in the hw counters section.

    • rp_cnp_handled
    • rp_cnp_ignored
    • np_cnp_sent
    • np_ecn_marked_roce_packets

     

    2. link_down_reason debug status was added.

     

    3. RDMA Diagnostics counters were added, see the list below.

     

    MLNX_OFED 4.0 Updates

    Note: the list of counters_ext was merged into port counters starting from MLNX_OFED 4.0. The folder was removed.

     

    Overview

    The counters discussed in this section relevant for both InifiniBand and Ethernet. Most originated from the InfiniBand spec, but relevant also when the interface is configured as Ethernet.

    For ethernet counters you can refer to Understanding mlx5 ethtool Counters for full list of counters.

     

    The counters are located under /sys/class/infiniband/ path as follows:

     

    Counter Groups

    There are two sets of counters

    1. Port Counters under the counters folder

    2. HW counters, under the hw_counters folder

    3. Debug Counters

     

    Port Counters

    # ll /sys/class/infiniband/mlx5_0/ports/1/counters

    total 0

    -r--r--r-- 1 root root 4096 Aug 29 12:10 excessive_buffer_overrun_errors

    -r--r--r-- 1 root root 4096 Aug 29 12:10 link_downed

    -r--r--r-- 1 root root 4096 Aug 29 12:10 link_error_recovery

    -r--r--r-- 1 root root 4096 Aug 29 12:10 local_link_integrity_errors

    -r--r--r-- 1 root root 4096 Aug 29 12:10 port_rcv_constraint_errors

    -r--r--r-- 1 root root 4096 Aug 29 12:10 port_rcv_data

    -r--r--r-- 1 root root 4096 Aug 29 12:10 port_rcv_errors

    -r--r--r-- 1 root root 4096 Aug 29 12:10 port_rcv_packets

    -r--r--r-- 1 root root 4096 Aug 29 12:10 port_rcv_remote_physical_errors

    -r--r--r-- 1 root root 4096 Aug 29 12:10 port_rcv_switch_relay_errors

    -r--r--r-- 1 root root 4096 Aug 29 12:10 port_xmit_constraint_errors

    -r--r--r-- 1 root root 4096 Aug 29 12:10 port_xmit_data

    -r--r--r-- 1 root root 4096 Aug 29 12:10 port_xmit_discards

    -r--r--r-- 1 root root 4096 Aug 29 12:10 port_xmit_packets

    -r--r--r-- 1 root root 4096 Aug 29 12:10 symbol_error

    -r--r--r-- 1 root root 4096 Aug 29 12:10 VL15_dropped

     

    HW Counters (RDMA diagnostics)

    # ls /sys/class/infiniband/mlx5_0/ports/1/hw_counters/ -al

    total 0

    -r--r--r-- 1 root root 4096 May 19 15:28 cq_overflow

    -r--r--r-- 1 root root 4096 May 19 15:28 duplicate_request

    -r--r--r-- 1 root root 4096 May 19 15:28 implied_nak_seq_err

    -rw-r--r-- 1 root root 4096 May 19 15:28 lifespan

    -r--r--r-- 1 root root 4096 May 19 15:28 local_ack_timeout_err

    -r--r--r-- 1 root root 4096 May 19 15:28 local_operation_error

    -r--r--r-- 1 root root 4096 May 19 15:28 np_cnp_sent

    -r--r--r-- 1 root root 4096 May 19 15:28 np_ecn_marked_roce_packets

    -r--r--r-- 1 root root 4096 May 19 15:28 out_of_buffer

    -r--r--r-- 1 root root 4096 May 19 15:28 out_of_sequence

    -r--r--r-- 1 root root 4096 May 19 15:28 packet_seq_err

    -r--r--r-- 1 root root 4096 May 19 15:28 req_bad_response

    -r--r--r-- 1 root root 4096 May 19 15:28 req_cqe_error

    -r--r--r-- 1 root root 4096 May 19 15:28 req_cqe_flush_error

    -r--r--r-- 1 root root 4096 May 19 15:28 req_local_length_error

    -r--r--r-- 1 root root 4096 May 19 15:28 req_local_protection

    -r--r--r-- 1 root root 4096 May 19 15:28 req_mw_binding

    -r--r--r-- 1 root root 4096 May 19 15:28 req_remote_access_errors

    -r--r--r-- 1 root root 4096 May 19 15:28 req_remote_invalid_request

    -r--r--r-- 1 root root 4096 May 19 15:28 req_remote_operation_errors

    -r--r--r-- 1 root root 4096 May 19 15:28 req_transport_retries_exceeded

    -r--r--r-- 1 root root 4096 May 19 15:28 resp_cqe_error

    -r--r--r-- 1 root root 4096 May 19 15:28 resp_cqe_flush_error

    -r--r--r-- 1 root root 4096 May 19 15:28 resp_local_length_error

    -r--r--r-- 1 root root 4096 May 19 15:28 resp_local_protection

    -r--r--r-- 1 root root 4096 May 19 15:28 resp_local_qp_error

    -r--r--r-- 1 root root 4096 May 19 15:28 resp_remote_access_errors

    -r--r--r-- 1 root root 4096 May 19 15:28 resp_remote_invalid_request

    -r--r--r-- 1 root root 4096 May 19 15:28 rnr_nak_retry_err

    -r--r--r-- 1 root root 4096 May 19 15:28 rp_cnp_handled

    -r--r--r-- 1 root root 4096 May 19 15:28 rp_cnp_ignored

    -r--r--r-- 1 root root 4096 May 19 15:28 rx_atomic_requests

    -r--r--r-- 1 root root 4096 May 19 15:28 rx_dct_connect

    -r--r--r-- 1 root root 4096 May 19 15:28 rx_read_requests

    -r--r--r-- 1 root root 4096 May 19 15:28 rx_write_requests

     

    Counter Description

    Port Counters Description

    CounterDescriptionInfiniBand Spec Name
    Group

    port_rcv_data

    Total number of data octets, divided by 4 (lanes), received on all VLs. This is 64 bit counter.

    PortRcvDataInformative

    port_rcv_packets

    Total number of packets (this may include packets containing Errors. This is 64 bit counter.

    PortRcvPktsInformative
    port_multicast_rcv_packets

    Total number of multicast packets, including multicast packets containing errors.

    PortMultiCastRcvPktsInformative
    port_unicast_rcv_packets

    Total number of unicast packets, including unicast packets containing errors.

    PortUnicastRcvPktsInformative

    port_xmit_data

    Total number of data octets, divided by 4 (lanes), transmitted on all VLs. This is 64 bit counter.

    PortXmitDataInformative

    port_xmit_packets

    port_xmit_packets_64

    Total number of packets transmitted on all VLs from this port. This may include packets with errors.

    This is 64 bit counter.

    PortXmitPktsInformative
    port_rcv_switch_relay_errorsTotal number of packets received on the port that were discarded because they could not be forwarded by the switch relay.PortRcvSwitchRelayErrorsError
    port_rcv_errorsTotal number of packets containing an error that were received on the port.PortRcvErrorsInformative
    port_rcv_constraint_errorsTotal number of packets received on the switch physical port that are discarded.PortRcvConstraintErrorsError
    local_link_integrity_errorsThe number of times that the count of local physical errors exceeded the threshold specified by LocalPhyErrors.LocalLinkIntegrityErrorsError
    port_xmit_waitThe number of ticks during which the port  had data to transmit but no data was sent during the entire tick (either because of insufficient credits or because of lack of arbitration).PortXmitWaitInformative
    port_multicast_xmit_packetsTotal number of multicast packets transmitted on all VLs from the port. This may include multicast packets with errors.PortMultiCastXmitPktsInformative
    port_unicast_xmit_packetsTotal number of unicast packets transmitted on all VLs from the port. This may include unicast packets with errors.PortUnicastXmitPktsInformative
    port_xmit_discards

    Total number of outbound packets discarded by the port because the port is down or congested.

    PortXmitDiscardsError
    port_xmit_constraint_errorsTotal number of packets not transmitted from the switch physical port.PortXmitConstraintErrorsError
    port_rcv_remote_physical_errorsTotal number of packets marked with the EBP delimiter received on the port.PortRcvRemotePhysicalErrorsError
    symbol_errorTotal number of minor link errors detected on one or more physical lanes.SymbolErrorCounterError
    VL15_droppedNumber of incoming VL15 packets dropped due to resource limitations (e.g., lack of buffers) of the port.VL15DroppedError
    link_error_recoveryTotal number of times the Port Training state machine has successfully completed the link error recovery process.LinkErrorRecoveryCounterError
    link_downedTotal number of times the Port Training state machine has failed the link error recovery process and downed the link.LinkDownedCounterError

     

     

    HW Counters Description

    CounterDescriptionGroup
    cq_overflow

    Number of times this Q tried to generate a CQE and failed due to full CQ.

    The counters was added in MLNX_OFED 4.1

    Error

    duplicate_request

    Number of received packets. A duplicate request is a request that had been previously executed.

    Error

    implied_nak_seq_err

    Number of time the requested decided an ACK. with a PSN larger than the expected PSN for an RDMA read or response.Error
    lifespan

     

    local_ack_timeout_err

    The number of No ACK response received from the responser within timer inteval, The QP retry limit was not excceed.

    Error
    local_operation_error

    Number of times requester detected local QP operation errors.

    The counters was added in MLNX_OFED 4.1

    Error

    out_of_buffer

    The number of drops occurred due to lack of WQE for the associated QPs.

    Error

    out_of_sequence

    The number of out of sequence packets received.

    Error
    packet_seq_errThe number of received NAK sequence  error packets. The QP retry limit was not exceeded.Error
    rnr_nak_retry_errThe number of received RNR NAK packets. The QP retry limit was not exceeded.Error
    rx_atomic_requestsThe number of received ATOMIC request for the associated QPs.Informative
    rx_dct_connectThe number of received connection request for the associated DCTs.Informative
    rx_read_requestsThe number of received READ requests for the associated QPs.Informative
    rx_write_requestsThe number of received WRITE requests for the associated QPs.Informative
    rp_cnp_handled

    The number of CNP packets handled by the Reaction Point HCA to throttle the transmission rate.

    The counters was added in MLNX_OFED 4.1

    Informative
    rp_cnp_ignored

    The number of CNP packets received and ignored by the Reaction Point HCA. This counter should not raise if RoCE Congestion Control was enabled in the network. If this counter raise, verify that ECN was enabled on the adapter. See HowTo Configure DCQCN (RoCE CC) for ConnectX-4 (Linux).

    The counters was added in MLNX_OFED 4.1

    Error
    np_cnp_sent

    The number of CNP packets sent by the Notification Point when it noticed congestion experienced in the RoCEv2 IP header (ECN bits).

    The counters was added in MLNX_OFED 4.1

    Informative
    np_ecn_marked_roce_packets

    The numver of RoCEv2 packets received by the notification point which were marked for experiencing the congestion (ECN bits where '11' on the ingress RoCE traffic) .

    The counters was added in MLNX_OFED 4.1

    Informative
    req_bad_response

    The number of times requester detected bad responses

    The counters was added in MLNX_OFED 4.1

    Error
    req_cqe_error

    The number of times requester detected CQEs completed with errors.

    The counters was added in MLNX_OFED 4.1

    Error
    req_cqe_flush_error

    The number of times requester detected CQEs completed with flushed errors.

    The counters was added in MLNX_OFED 4.1

    Error
    req_local_length_error

    The number of times requester detected local length errors.

    The counters was added in MLNX_OFED 4.1

    Error
    req_local_protection

    The number of times requester detected memory protection errors.

    The counters was added in MLNX_OFED 4.1

    Error
    req_mw_binding

    The number of times requester detected memory window binding errors.

    The counters was added in MLNX_OFED 4.1

    Error
    req_remote_access_errors

    The number of times requester detected remote access errors.

    The counters was added in MLNX_OFED 4.1

    Error
    req_remote_invalid_request

    The number of times requester detected remote invalid request errors.

    The counters was added in MLNX_OFED 4.1

    Error
    req_remote_operation_errors

    The number of times requester detected remote operation errors.

    The counters was added in MLNX_OFED 4.1

    Error
    req_transport_retries_exceeded

    The number of times requester detected  transport retries exceed errors.

    The counters was added in MLNX_OFED 4.1

    Error
    resp_cqe_error

    The number of times responder detected CQEs completed with errors.

    The counters was added in MLNX_OFED 4.1

    Error
    resp_cqe_flush_error

    The number of times responder detected CQEs completed with flushed errors.

    The counters was added in MLNX_OFED 4.1

    Error
    resp_local_length_error

    The number of times responder detected local length errors.

    The counters was added in MLNX_OFED 4.1

    Error
    resp_local_protection

    The number of times responder detected memory protection errors.

    The counters was added in MLNX_OFED 4.1

    Error
    resp_local_qp_error

    The number of times responder detected local QP operation errors.

    The counters was added in MLNX_OFED 4.1

    Error
    resp_remote_access_errors

    The number of times responder detected remote access errors.

    The counters was added in MLNX_OFED 4.1

    Error
    resp_remote_invalid_request

    The number of times responder detected remote invalid request errors.

    The counters was added in MLNX_OFED 4.1

    Error

     

     

    Debug Status Parameters

    The following parameters are located in /sys/class/net/<interface>/debug

     

    ParameterDescriptionDefault
    lro_timeout

    Sets the LRO timer period value in usecs which will be used as LRO session expiration time.

    Example:

    # cat /sys/class/net/eth2/debug/lro_timeout

    Actual timeout: 32

    Supported timeout: 8 16 32 1024

    32
    link_down_reason

    Link down reason will allow the user to query the reason which is preventing the link from going up.

    Example:

     

    $ cat /sys/class/net/ethXX/debug/link_down_reason

    monitor_opcode: 0x0

    status_message: The port is Active.

     

    Refer to the adapter PRM for all possible options (PDDR register).

     

    This parameter is available starting in MLNX_OFED 4.1

    N/A