4 Replies Latest reply on Apr 1, 2017 1:25 AM by black.s

    lustre, mpi and qos


      Hello all


      I want to setup qos for lustre and MPI.

      How I can setup it ?



      As I understand, you can configure in two ways:

      1) The link http://www.spinics.net/lists/linux-rdma/msg02150.html

      recommended use like this:

      in /etc/opensm/qos-policy.conf


              default :0                          # default SL (OPENMPI or MPI)

              lustre, service-id 0x : 1    # Lustre


      write to /etc/modprobe.d/lustre.conf

         options ko2iblnd port_space=339


      Where I can get service-id or how I can refer to lustre traffic in qos-policy.conf and setup it your own SL ?

      I do not want to use "targetportguid". It so direct.

      What is port_space=339 ? Why 339 or where i can get it ? How port_space connected to lustre and qos ?


      2) Can I direct setup lustre (OSS and MDS) your own SL ?

      May be it possibly in Environment variable or may be exist other ways ?



      If I using MVAPICH2 How I can setup SL for MPI traffic ? Is I can do it by Environment variable or may be exist other ways ?


      Sorry for my english

        • Re: lustre, mpi and qos

          Hi Serge,


          you can find an example for implementing what you need on the Mellanox OFED user guide: http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_User_Manual_v4.0.pdf , QoS chapter 9.7

            • Re: lustre, mpi and qos

              I read this manual. Sorry but I cant found answer in this manual and I cant found in this manual the chapter "QoS chapter 9.7". Manual have only six chapters.


              In page 154 QoS Configuration Examples

              "MPI is assigned an SL via the command line

              host1# mpirun –sl 0". I cant found this option in my mvapich2 2.1.


              In page 154 QoS Configuration Examples recommendation to setup qos-ulps that



              :0 # default SL (for


                any, target-port-guid OST1,OST2,OST3,OST4:1 # SL for Lustre OST

                any, target-port-guid MDS1,MDS2 :2 # SL for Lustre



              But it setup by target-port-guid. It so direct issues. I want to be use service-ID or some like this. More flexible.


              I also cant found about Lustre and Service-ID or some other ways to setup Lustre your SL in this manual.


              But still thanks for answer

            • Re: lustre, mpi and qos

              I found answers

              MPI - only for MVAPICH2 [mvapich-discuss] MPI with service level . We can use some enviroments

              mpirun_rsh -np <num_procs> <host_list>

                MV2_DEFAULT_SERVICE_LEVEL=<desired SL> <path to executable>

              Lustre: How I can understand work only qos_policy reference with target GUID.


              If anybody know other method for lustre then ping me.


              Thanks all