3 Replies Latest reply on Mar 12, 2015 1:32 AM by yairi

    How to tune mlx4_core various parameters found in /sys/module/mlx4_core/parameters for optimal performance

      How to tune Linux OFED mlx4_core various parameters found in /sys/module/mlx4_core/parameters for optimal performance of IB FDR and QDR adapters for memory intensive / high bandwidth MPI communication.

      Reference to the doc or information is appreciated.

          • Re: How to tune mlx4_core various parameters found in /sys/module/mlx4_core/parameters for optimal performance

            Yari,


            Thanks for the pointer. I looked at the the tuning guide, it is good, but targeted for ethernet over IB performance and doesn't delve much into details of IB driver settings, although there is a section about autotuning. 

             

            But you may have some more, see  the list of params here for kernel modules:

             

            We see that tuning/increasing the default params in /sys/module/mlx4_core/parameters resolve issues, we even see some performance boost for our use case. (when we increase log_num_mtt, log_mtts_per_seg).

            We really need to understand how to obtain max performance tuning these and other driver params. I doubt the auto-tuners would resolve it for us..

             

            ll /sys/module/mlx4_core/parameters/

            -rw-r--r-- 1 root root 4096 Mar 11 17:23 block_loopback

            -rw-r--r-- 1 root root 4096 Mar 11 17:23 debug_level

            -rw-r--r-- 1 root root 4096 Mar 11 17:23 enable_pre_t11_mode

            -r--r--r-- 1 root root 4096 Mar 11 17:23 enable_qinq

            -r--r--r-- 1 root root 4096 Mar 11 17:23 enable_qos

            -r--r--r-- 1 root root 4096 Mar 11 17:23 fast_drop

            -r--r--r-- 1 root root 4096 Mar 11 17:23 high_rate_steer

            -rw-r--r-- 1 root root 4096 Mar 11 17:23 internal_err_reset

            -r--r--r-- 1 root root 4096 Mar 11 17:23 log_mtts_per_seg

            -r--r--r-- 1 root root 4096 Mar 11 17:23 log_num_cq

            -r--r--r-- 1 root root 4096 Mar 11 17:23 log_num_mac

            -r--r--r-- 1 root root 4096 Mar 11 17:23 log_num_mcg

            -r--r--r-- 1 root root 4096 Mar 11 17:23 log_num_mgm_entry_size

            -r--r--r-- 1 root root 4096 Mar 11 17:23 log_num_mpt

            -r--r--r-- 1 root root 4096 Mar 11 17:23 log_num_mtt

            -r--r--r-- 1 root root 4096 Mar 11 17:23 log_num_qp

            -r--r--r-- 1 root root 4096 Mar 11 17:23 log_num_srq

            -r--r--r-- 1 root root 4096 Mar 11 17:23 log_rdmarc_per_qp

            -r--r--r-- 1 root root 4096 Mar 11 17:23 msi_x

            -r--r--r-- 1 root root 4096 Mar 11 17:23 pfcrx

            -r--r--r-- 1 root root 4096 Mar 11 17:23 pfctx

            -r--r--r-- 1 root root 4096 Mar 11 17:23 probe_vf

            -r--r--r-- 1 root root 4096 Mar 11 17:23 set_4k_mtu

            -r--r--r-- 1 root root 4096 Mar 11 17:23 sr_iov

            -r--r--r-- 1 root root 4096 Mar 11 17:23 use_prio

             

             

            ll /sys/module/mlx4_en/parameters

            total 0

            -r--r--r-- 1 root root 4096 Mar 11 17:40 enable_sys_tune

            -r--r--r-- 1 root root 4096 Mar 11 17:40 inline_thold

            -r--r--r-- 1 root root 4096 Mar 11 17:40 mem_node

            -r--r--r-- 1 root root 4096 Mar 11 17:40 num_rx_rings

            -r--r--r-- 1 root root 4096 Mar 11 17:40 udp_rss

             

            Thanks,
            Mous.

          • Re: How to tune mlx4_core various parameters found in /sys/module/mlx4_core/parameters for optimal performance
            yairi

            Hi Mous,

            the list of parameters you present is applicable for various modes and applications.

            The Tuning guide i provided you is not only for IPoIB, it is also more general purposes IB only applications (That uses RDMA). One god example is the log_num_mtt setting.

            I am not really familiar with your application and what type of resources it specifically need, You can either open a ticket with Mellanox support or expend more here about what exactly are you looking to do.

             

             

            Thanks!