What's New in the MLNX_OFED v4.1 Release

Version 32

    This post highlights the features of MLNX_OFED v4.1.

     

     

    References

     

    Download Driver

    Download the MLNX_OFED driver from Mellanox web page, click here.

     

    Tools

    mlx_fs_dump

    mlx_fs_dump tool is introduced in this release. This is a python tool that prints the steering rules in a readable manner. For basic usage and examples, seemlx_fs_dump.

     

    Virtualization

    Probed VFs

    Probed VFs (mlx5): Sometimes it is not desirable to probe the virtual functions after SR-IOV is enabled. This can save host side resource usage by VF instances, which would eventually be probed to the VMs. It might be very inconvenient if there are a lot of VFs created and probed in the hypervisor. A new mlx5_core module parameter called probe_vf was added to provide this option.

    For more information, see HowTo Configure and Probe VFs on mlx5 Drivers.

     

    Monitoring

    ECN Congestion Counters

    Four ECN/CNP Congestion counters were added to mlx5 driver in this release. For more information, see Understanding mlx5 Linux Counters and Status Parameters.

     

    The reason for Link DOWN was added to mlx5 drive. link_down_reason debug status was added. For more information, see Understanding mlx5 Linux Counters and Status Parameters.

     

    RDMA Diagnostic Counters

    Various RDMA Diagnostic counters were added to the following path:

    ls /sys/class/infiniband/mlx5_0/ports/1/hw_counters/

    For more information, see Understanding mlx5 Linux Counters and Status Parameters

     

    Ethtool

    One of the features in ethtool is rx-fcs. Normally, the FCS of the packet will be truncated by the ASIC hardware before sending it to the application socket buffer (skb). Ethtool allows setting the rx-fcs not to be truncated, but to pass it to the application for analysis. For more information,see Understanding ethtool rx-fcs for mlx5 Drivers.

     

    RDMA and RoCE

     

    DSCP Trust

    This release supports DSCP in Trust mode. Users now have the ability to enable PFC based on the DSCP value. With this solution, VLAN header usage is no longer mandatory.

    To learn more about DSCP Trust mode and for a configuration example, see HowTo Configure Trust Mode on Mellanox Adapters.

     

    ECN Parameters

    ECN parameters were moved to a different location. All the parameters are now located in:

    /sys/kernel/debug/mlx5/<PCI BUS>/cc_params/

    In addition, few parameters were removed, and one parameter was added. For more information, see HowTo Configure DCQCN (RoCE CC) for ConnectX-4 (Linux).

     

    RDMA CM Default RoCE Mode

    Starting with MLNX_OFED 4.1, the RDMA CM uses RoCEv2 for the supported adapters (ConnectX-3 Pro or later)  by default. When testing RDMA CM, it is recommended to update the client and server to MLNX_OFED 4.1.

    For more information, see RDMA CM and RoCE Version Defaults.

     

    Storage

    NVMe-oF Target offload

    NVMe-oF target offload is supported in ConnectX-5 adapters. To enable it, see HowTo Configure NVMe-oF Target Offload.

    For testing your storage server and displaying the gained utilization, see Simple NVMe-oF Target Offload Benchmark.

     

    Programming

    PeerDirect Programming

    New experimental verb "ibv_exp_open_device" was added to OFED v4.1 (on ConnectX-4 adapters). This verb adds the ability to open a device and create a context while giving the PCI peer attributes (in ibv_exp_open_device_attr) such as name and ID. For more information and coding examples, see PeerDirect Programming: Create Context with Peer Attributes.

     

    InfiniBand

    PTP over IPoIB

    Added support for PTP feature in IPoIB offloaded devices. This feature allows for accurate synchronization between the distributed entities over the network. The synchronization is based on symmetric Round Trip

    Time (RTT) between the master and slave devices. The feature is enabled by default.

    For more information, see Running Linux PTP with ConnectX-4.

     

    IPoIB Improvements

    IPoIB interface was improved and optimized with the following hardware features for ConnectX-4 adapter cards:

    • Stateless offloads (LRO, aRFS, RSS, TSS)
    • Multi-queue support. All queues share the same transport
    • Interrupt moderation
    • Multi partitions optimizations
    • Share send/recv Work Queues
    • Vendor specific optimizations
    • Work queue processing
    • The IPoIB ULP was leveraged while the support of the “Legacy” mode remains.
    • Support UD mode only

     

    A simple way to verify that the driver is using the new mode by shell command: “ip link show ibX” the driver mac will start with 00:xxxxxxx

    No new configuration added, all enabled by default and done internally within the IPoIB driver.

     

    Time Synchronization

    1PPS

    One Pulse Per Second (1PPS) is a time synchronization feature that allows the adapter to be able to send or receive 1 pulse per second on a dedicated pin on the adapter card.

    The supported ASICs are ConnectX-4 or later. Only one pin is supported and could be configured as 1PPS in or 1PPS out. For testing, a special adapter should be used (with access to the 1PPS pin) - contact Mellanox support.

    To learn how to test it, see HowTo Test 1PPS on Mellanox Adapters.

     

    Additional information on the content of MLNX_OFED release can be found in MLNX_OFED Release Notes document available on Mellanox official website.