ConnectX-3/Pro Tuning For Linux (Idle Loop and IP Forwarding)

Version 1

    This post shows specific tuning for ConnectX-3/Pro (mlx4 driver). This post is not relevant for ConnectX-4/5 (mlx5) adapter cards.

     

    References

    Performance Tuning for Mellanox Adapters

     

    Tuning Options

    Kernel Idle Loop Tuning

     

    The mlx4_en kernel module has an optional parameter that can tune the kernel idle loop for better latency. This will improve the CPU wake-up time but may result in higher power consumption. To tune the kernel idle loop, set the following options in the /etc/modprobe.d/mlx4_core.conf file:

    Note: if the file does not exist, is must be created having the same name as the one stated above.

     

    options mlx4_core enable_sys_tune=1

     

    Multi-Threaded IP Forwarding Tuning

    To optimize NIC usage as IP forwarding:

    1. Set the following options in /etc/modprobe.d/mlx4.conf.

    options mlx4_en inline_thold=0

    options mlx4_core log_num_mgm_entry_size=-7

     

    2. Optimize steering, set the following options in /etc/modprobe.d/mlx4.conf.

    options mlx4_en inline_thold=0

    options mlx4_core log_num_mgm_entry_size=-7

     

    3. Restart the driver

     

    4. Apply interrupt affinity tuning, forwarding on the same interface:

    # set_irq_affinity_bynode.sh <numa node> <interface>  

     

    5. Disable adaptive interrupt moderation and set status values, using:

    # ethtool -C <interface> adaptive-rx off rx-usecs 0 tx-frames 64