5 Replies Latest reply on Nov 2, 2015 12:31 PM by sophie

    Read port priority counters is 0 .

    raincaller

      Hi.

       

      I had a problem port priority  3 counters is not.

       

      Reference is https://community.mellanox.com/docs/DOC-1414

       

      What should we check

       

      Thank you in advance!

       

       

       

      pc1 - Switch - pc2

       

      Switch firmware  3.4.2008

       

      pc1 and 2 is Ububtu 14.04 

      Fimware 2.35.5100

      Driver 3.1.-1.0.3

       

       

      pc1

       

      root@Clotho:~# ib_write_bw -R --report_gbits --port=12500 -D 10

       

       

      ************************************

      * Waiting for client to connect... *

      ************************************

      ---------------------------------------------------------------------------------------

                          RDMA_Write BW Test

      Dual-port       : OFF          Device         : mlx4_0

      Number of qps   : 1            Transport type : IB

      Connection type : RC           Using SRQ      : OFF

      CQ Moderation   : 100

      Mtu             : 1024[B]

      Link type       : Ethernet

      Gid index       : 0

      Max inline data : 0[B]

      rdma_cm QPs     : ON

      Data ex. method : rdma_cm

      ---------------------------------------------------------------------------------------

      Waiting for client rdma_cm QP to connect

      Please run the same command with the IB/RoCE interface IP

      ---------------------------------------------------------------------------------------

      local address: LID 0000 QPN 0x025a PSN 0x86a3f8

      GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:10

      remote address: LID 0000 QPN 0x025a PSN 0x35cd7d

      GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:20

      ---------------------------------------------------------------------------------------

      #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]

      65536      417600           0.00               36.47              0.069568

      ---------------------------------------------------------------------------------------

       

      root@Clotho:~# ethtool -S p1p1 | grep prio_3

           rx_pause_prio_3: 0

           rx_pause_duration_prio_3: 0

           rx_pause_transition_prio_3: 0

           tx_pause_prio_3: 0

           tx_pause_duration_prio_3: 0

           tx_pause_transition_prio_3: 0

           rx_prio_3_packets: 0

           rx_prio_3_bytes: 0

           tx_prio_3_packets: 0

           tx_prio_3_bytes: 0

       

      root@Clotho:~# ./roce-check.sh

      ============================

        pfcrx / pfctx parameters 

      ============================

      INTERFACE: mlx4_en

        RX: 0x08

        TX: 0x08

      ============================

        VLAN parameters          

      ============================

      VLAN: p1p1.100

      EGRESS priority mappings: 0:3 1:3 2:3 3:3 4:3 5:3 6:3 7:3

       

       

      pc2.

       

      ib_write_bw -R --report_gbits 192.168.10.10  --port=12500 -D 10

      ---------------------------------------------------------------------------------------

                          RDMA_Write BW Test

      Dual-port       : OFF          Device         : mlx4_0

      Number of qps   : 1            Transport type : IB

      Connection type : RC           Using SRQ      : OFF

      TX depth        : 128

      CQ Moderation   : 100

      Mtu             : 1024[B]

      Link type       : Ethernet

      Gid index       : 0

      Max inline data : 0[B]

      rdma_cm QPs     : ON

      Data ex. method : rdma_cm

      ---------------------------------------------------------------------------------------

      local address: LID 0000 QPN 0x025a PSN 0x35cd7d

      GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:20

      remote address: LID 0000 QPN 0x025a PSN 0x86a3f8

      GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:10

      ---------------------------------------------------------------------------------------

      #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]

      Conflicting CPU frequency values detected: 2799.937000 != 1220.531000. CPU Frequency is not max.

      65536      417600           0.00               36.47              0.069568

      ---------------------------------------------------------------------------------------

       

       

      root@Aprotos:~# ethtool -S p1p1 | grep prio_3

           rx_pause_prio_3: 0

           rx_pause_duration_prio_3: 0

           rx_pause_transition_prio_3: 0

           tx_pause_prio_3: 0

           tx_pause_duration_prio_3: 0

           tx_pause_transition_prio_3: 0

           rx_prio_3_packets: 0

           rx_prio_3_bytes: 0

           tx_prio_3_packets: 0

           tx_prio_3_bytes: 0

      root@Aprotos:~# ./roce-check.sh

      ============================

        pfcrx / pfctx parameters 

      ============================

      INTERFACE: mlx4_en

        RX: 0x08

        TX: 0x08

      ============================

        VLAN parameters          

      ============================

      VLAN: p1p1.100

      EGRESS priority mappings: 0:3 1:3 2:3 3:3 4:3 5:3 6:3 7:3

        • Re: Read port priority counters is 0 .
          ophirmaor

          Hi,

          1.Can you send the output on the switch for the port priority counters?

          2. Can you send the running config of the switch, to make sure PFC is enabled.

          3. Which adapter is used?

          4. can you send the output for this

          RX=`cat /sys/module/mlx4_en/parameters/pfcrx`;printf "0x%x\n" $RX

          and

          TX=`cat /sys/module/mlx4_en/parameters/pfctx`;printf "0x%x\n" $TX

            • Re: Read port priority counters is 0 .
              raincaller

              Thanks, Ophir Maor

               

              1. Switch Port 1 and 5 not packets counters

                 

              switch-5e13ce [standalone: master] (config) # show interfaces ethernet 1/1 counters priority 3

               

              Rx

                0                    packets

                0                    unicast packets

                0                    multicast packets

                0                    broadcast packets

                0                    bytes

                0                    pause packets

                0                    pause duration milliseconds

               

              Tx

                0                    packets

                0                    unicast packets

                0                    multicast packets

                0                    broadcast packets

                0                    bytes

                0                    pause packets

               

              switch-5e13ce [standalone: master] (config) # show interfaces ethernet 1/1 counters priority 5

               

              Rx

                0                    packets

                0                    unicast packets

                0                    multicast packets

                0                    broadcast packets

                0                    bytes

                0                    pause packets

                0                    pause duration milliseconds

               

              Tx

                0                    packets

                0                    unicast packets

                0                    multicast packets

                0                    broadcast packets

                0                    bytes

                0                    pause packets

               

              2. yes, Switch  PFC is enabled

               

              ---- switch running config --

              ##

              ## Running database "initial"

              ## Generated at 2015/10/30 11:19:07 +0000

              ## Hostname: switch-5e13ce

              ##

               

               

              ##

              ## Running-config temporary prefix mode setting

              ##

              no cli default prefix-modes enable

               

               

              ##

              ## License keys

              ##

                 license install ..........................................

                

              ##

              ## DCBX PFC configuration

              ##

                 dcb priority-flow-control enable force

                 dcb priority-flow-control priority 3 enable

                 interface ethernet 1/1 dcb priority-flow-control mode on force

                 interface ethernet 1/5 dcb priority-flow-control mode on force

                

              ##

              ## Interface Ethernet configuration

              ##

                 interface ethernet 1/1 mtu 4200 force

                 interface ethernet 1/5 mtu 4200 force

                 interface ethernet 1/1 switchport mode hybrid

                 interface ethernet 1/5 switchport mode hybrid

                

              ##

              ## VLAN configuration

              ##

                 vlan 100

                

                 interface ethernet 1/1 switchport hybrid allowed-vlan all

                 interface ethernet 1/5 switchport hybrid allowed-vlan all

                

              ##

              ## DCBX ETS configuration

              ##

              no dcb ets enable

                 dcb ets tc bandwidth 20 20 30 30

                

              ##

              ## Network interface configuration

              ##

              no interface mgmt0 dhcp

                 interface mgmt0 ip address xxx.xxx.xx.xxx /25

                

              ##

              ## Network interface IPv6 configuration

              ##

              no interface mgmt0 ipv6 address autoconfig default

              no interface mgmt0 ipv6 enable

                

              ##

              ## Other IP configuration

              ##

                 ip route 0.0.0.0 0.0.0.0 xxx.xxx.xxx.xxx

                

              ##

              ## AAA remote server configuration

              ##

              # ldap bind-password ********

              # radius-server key ********

              # tacacs-server key ********

                

              ##

              ## Network management configuration

              ##

              # web proxy auth basic password ********

                

              ##

              ## X.509 certificates configuration

              ##

              #

              # Certificate name system-self-signed, ID 0e5aa8139ab66157a7a051bcf66540771fb3cab2

              # (public-cert config omitted since private-key config is hidden)

               

               

              ##

              ## Persistent prefix mode setting

              ##

              cli default prefix-modes enable

               

              3. My FDR HCA is MCX353A-FCBT (both)

               

              4. RX=`cat /sys/module/mlx4_en/parameters/pfcrx`;printf "0x%x\n" $RX

                  TX=`cat /sys/module/mlx4_en/parameters/pfctx`;printf "0x%x\n" $TX

               

              Result is

               

              pc1.

               

              root@Clotho:/etc/modprobe.d#  TX=`cat /sys/module/mlx4_en/parameters/pfctx`;printf "0x%x\n" $TX

              0x8

              root@Clotho:/etc/modprobe.d# RX=`cat /sys/module/mlx4_en/parameters/pfcrx`;printf "0x%x\n" $RX

              0x8

              root@Clotho:/etc/modprobe.d#

               

              pc2.

              root@Atropos:/etc/modprobe.d#   TX=`cat /sys/module/mlx4_en/parameters/pfctx`;printf "0x%x\n" $TX

              0x8

              root@Atropos:/etc/modprobe.d# RX=`cat /sys/module/mlx4_en/parameters/pfcrx`;printf "0x%x\n" $RX

              0x8

              root@Atropos:/etc/modprobe.d#

               

               

               

              And See most of the reference to the document, but The Result is still the same.

               

              Any help Thanks!!

            • Re: Read port priority counters is 0 .
              sophie

              Hi Seol,

              1) MLNX-OS Switch Configuration:
                 a. Have you created a VLAN and set the switchports to hybrid (or trunk) mode?
                 b. Have you Enable PFC on the switch and on the interfaces?
                 c. Does the "show dcb priority-flow-control" reports the accurate configuration?
                
              2) Server configuration:
                 a. Do you have the /etc/modprobe.d/mlx4_en.conf file with the following options:
                    options mlx4_en pfctx=0x08 pfcrx=0x08 
                 b. Did you restart openibd service ? (#/etc/init.d/openidb restart)
                    Please also verify under /sys/module/mlx4_en/parameters/pfcrx & pfctx files that the priority is set accordingly.
                 c. When you run the "ib_write_bw" from the client (Aprotos), does the IP 192.168.10.10 correspond to the VLAN: p1p1.100 on server Clotho? 

              3) Port priority counters from the Mellanox switch (via MLNX-OS CLI):
                 a. Does the interface(s) on the switch report any pause packets? (check for both interfaces from the servers).
                    ie: switch (config) # show interfaces ethernet 1/1 counters priority 3
                
                 Thank you,
                 Sophie.

                • Re: Read port priority counters is 0 .
                  raincaller

                  Thanks, Sophie Naudin

                   

                  1. Swich configuration is Hybrid mode and enable pfc setting seems there is no problem.

                   

                   

                  switch-5e13ce [standalone: master] (config) # show dcb priority-flow-control

                   

                  PFC enabled

                  Priority Enabled List    :3

                  Priority Disabled List   :0 1 2 4 5 6 7

                   

                  TC     Lossless

                  ---    ----------

                  0           N

                  1           Y

                  2           Y

                  3           N

                   

                  Interface      PFC admin        PFC oper

                  ------------   --------------   -------------

                  Eth1/1           On               Enabled

                  Eth1/2           Disabled         Disabled

                  Eth1/3           Disabled         Disabled

                  Eth1/4           Disabled         Disabled

                  Eth1/5           On               Enabled

                  Eth1/6           Disabled         Disabled

                  Eth1/7           Disabled         Disabled

                  Eth1/8           Disabled         Disabled

                  Eth1/9           Disabled         Disabled

                  Eth1/10          Disabled         Disabled

                  Eth1/11          Disabled         Disabled

                  Eth1/12          Disabled         Disabled

                  Eth1/13          Disabled         Disabled

                  Eth1/14          Disabled         Disabled

                  Eth1/15          Disabled         Disabled

                  Eth1/16          Disabled         Disabled

                  Eth1/17          Disabled         Disabled

                  Eth1/18          Disabled         Disabled

                  Eth1/19          Disabled         Disabled

                  Eth1/20          Disabled         Disabled

                  Eth1/21          Disabled         Disabled

                  Eth1/22          Disabled         Disabled

                  Eth1/23          Disabled         Disabled

                  Eth1/24          Disabled         Disabled

                  Eth1/25          Disabled         Disabled

                  Eth1/26          Disabled         Disabled

                  Eth1/27          Disabled         Disabled

                  Eth1/28          Disabled         Disabled

                  Eth1/29          Disabled         Disabled

                  Eth1/30          Disabled         Disabled

                  Eth1/31          Disabled         Disabled

                  Eth1/32          Disabled         Disabled

                  Eth1/33          Disabled         Disabled

                  Eth1/34          Disabled         Disabled

                  Eth1/35          Disabled         Disabled

                  Eth1/36          Disabled         Disabled

                   

                  2. According to your instruction Check it all.

                  mlx4_en.conf

                  openibd restart

                  /sys/module/mlx4_en/parameters/pfcrx & pfctx

                   

                  pc1

                   

                  root@Atropos:~# cat /proc/net/vlan/p1p1.100

                  p1p1.100  VID: 100       REORDER_HDR: 1  dev->priv_flags: 1

                           total frames received            0

                             total bytes received            0

                    Broadcast/Multicast Rcvd            0

                   

                   

                        total frames transmitted            8

                         total bytes transmitted          648

                  Device: p1p1

                  INGRESS priority mappings: 0:0  1:0  2:0  3:0  4:0  5:0  6:0 7:0

                  EGRESS priority mappings: 0:3 1:3 2:3 3:3 4:3 5:3 6:3 7:3

                   

                  root@Atropos:~# ib_write_bw -R --report_gbits --port=12500 -D 10

                   

                   

                  ************************************

                  * Waiting for client to connect... *

                  ************************************

                  ---------------------------------------------------------------------------------------

                                      RDMA_Write BW Test

                  Dual-port       : OFF          Device         : mlx4_0

                  Number of qps   : 1            Transport type : IB

                  Connection type : RC           Using SRQ      : OFF

                  CQ Moderation   : 100

                  Mtu             : 2048[B]

                  Link type       : Ethernet

                  Gid index       : 0

                  Max inline data : 0[B]

                  rdma_cm QPs     : ON

                  Data ex. method : rdma_cm

                  ---------------------------------------------------------------------------------------

                  Waiting for client rdma_cm QP to connect

                  Please run the same command with the IB/RoCE interface IP

                  ---------------------------------------------------------------------------------------

                  local address: LID 0000 QPN 0x0267 PSN 0xfa896c

                  GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:20

                  remote address: LID 0000 QPN 0x026e PSN 0x5ce508

                  GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:30

                  ---------------------------------------------------------------------------------------

                  #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]

                  65536      447200           0.00               39.07              0.074512

                  ---------------------------------------------------------------------------------------

                   

                  pc2

                   

                  root@ubuntu:/etc/modprobe.d# cat /proc/net/vlan/p3p1.100

                  p3p1.100  VID: 100       REORDER_HDR: 1  dev->priv_flags: 1

                           total frames received            0

                             total bytes received            0

                    Broadcast/Multicast Rcvd            0

                   

                   

                        total frames transmitted            8

                         total bytes transmitted          648

                  Device: p3p1

                  INGRESS priority mappings: 0:0  1:0  2:0  3:0  4:0  5:0  6:0 7:0

                  EGRESS priority mappings: 0:3 1:3 2:3 3:3 4:3 5:3 6:3 7:3

                   

                  root@ubuntu:/etc/modprobe.d# ib_write_bw -R --report_gbits 192.168.10.20 --port=12500 -D 10

                  ---------------------------------------------------------------------------------------

                                      RDMA_Write BW Test

                  Dual-port       : OFF          Device         : mlx4_1

                  Number of qps   : 1            Transport type : IB

                  Connection type : RC           Using SRQ      : OFF

                  TX depth        : 128

                  CQ Moderation   : 100

                  Mtu             : 2048[B]

                  Link type       : Ethernet

                  Gid index       : 0

                  Max inline data : 0[B]

                  rdma_cm QPs     : ON

                  Data ex. method : rdma_cm

                  ---------------------------------------------------------------------------------------

                  local address: LID 0000 QPN 0x026e PSN 0x5ce508

                  GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:30

                  remote address: LID 0000 QPN 0x0267 PSN 0xfa896c

                  GID: 00:00:00:00:00:00:00:00:00:00:255:255:192:168:10:20

                  ---------------------------------------------------------------------------------------

                  #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]

                  Conflicting CPU frequency values detected: 1199.980000 != 3799.960000. CPU Frequency is not max.

                  65536      447200           0.00               39.07              0.074512

                  ---------------------------------------------------------------------------------------

                   

                  3.

                   

                  switch-5e13ce [standalone: master] (config) # show interfaces ethernet 1/1 counters priority 3

                   

                   

                  Rx

                    0                    packets

                    0                    unicast packets

                    0                    multicast packets

                    0                    broadcast packets

                    0                    bytes

                    0                    pause packets

                    0                    pause duration milliseconds

                   

                   

                  Tx

                    0                    packets

                    0                    unicast packets

                    0                    multicast packets

                    0                    broadcast packets

                    0                    bytes

                    0                    pause packets

                  switch-5e13ce [standalone: master] (config) # show interfaces ethernet 1/1 counters priority 0

                   

                   

                  Rx

                    176634481            packets

                    176634354            unicast packets

                    34                   multicast packets

                    93                   broadcast packets

                    697235467340         bytes

                    0                    pause packets

                    0                    pause duration milliseconds

                   

                   

                  Tx

                    164914925            packets

                    164907341            unicast packets

                    7583                 multicast packets

                    1                    broadcast packets

                    648406932028         bytes

                    0                    pause packets

                  switch-5e13ce [standalone: master] (config) # show interfaces ethernet 1/5 counters priority 3

                   

                   

                  Rx

                    0                    packets

                    0                    unicast packets

                    0                    multicast packets

                    0                    broadcast packets

                    0                    bytes

                    0                    pause packets

                    0                    pause duration milliseconds

                   

                   

                  Tx

                    0                    packets

                    0                    unicast packets

                    0                    multicast packets

                    0                    broadcast packets

                    0                    bytes

                    0                    pause packets

                  switch-5e13ce [standalone: master] (config) # show interfaces ethernet 1/5 counters priority 0

                   

                   

                  Rx

                    164907378            packets

                    164907341            unicast packets

                    36                   multicast packets

                    1                    broadcast packets

                    645108150520         bytes

                    0                    pause packets

                    0                    pause duration milliseconds

                   

                   

                  Tx

                    176641795            packets

                    176634354            unicast packets

                    7416                 multicast packets

                    25                   broadcast packets

                    700768765837         bytes

                    0                    pause packets

                  switch-5e13ce [standalone: master] (config) #

                   

                   

                  Still now it is not solve. Advise, Thanks!!

                    • Re: Read port priority counters is 0 .
                      sophie

                      Hi Seol,

                       

                      Originally you used PC1 "Clotho" & PC2 "Aprotos", on my previous statement I asked:

                       

                      1) When you run the "ib_write_bw" from the client (Aprotos), does the IP 192.168.10.10 correspond to the VLAN: p1p1.100 on server Clotho?

                      2) If you are testing with client (Clotho) to server (Aprotos), do the counters report the same?

                      3) Are you now introducing and testing with a different host? (Ubuntu)

                      4) What are the switch & HCA model/partnumber/PSID?

                       

                      Thank you,

                      Sophie.