1 Reply Latest reply on Mar 7, 2017 12:08 PM by alkx

    ConnectX-4 LX RoCE does not like latency

    rldleblanc

      This is a diagram of my current set up.

       

                      +----------------+

                      |  Linux Router  |

                      |   ConnectX-3   |

                      | port 1  port 2 |

                      +----------------+

                           /      \

      +---------------+   /        \   +---------------+

      |    Host 1     |  / A      A \  |    Host 2     |

      | ConnectX-4-LX | /            \ | ConnectX-4-LX |

      |        Port 1 |-              -| Port 1        |

      |        Port 2 |----------------| Port 2        |

      +---------------+        B       +---------------+

       

      The Linux router has the ConnectX-3 (not PRO) card in Ethernet mode and is using a breakout cable (port 1 only) to connect to the ConnectX-4-LX cards at 10 Gb as path 'A'. The second port of the ConnectX-4-LX cards are connected directly at 25 Gb as path 'B'. Host 1 & 2 are running CentOS 7.2 with 3.10.0-327.36.3.el7.x86_64 and OFED 3.4. Linux router is running CentOS 7.2 with 4.9.0 kernel.

       

      Iser and RDMA works fine over path 'B' and path 'A' (in either bridge or router mode) and now I want to add latency and drop packets to understand the effects. I'm using tc and netem to add the latency into the path. When I add .5 ms of latency in both directions, iSER slows to a crawl, throws errors in dmesg and sometimes even causes the file system to go read only. If I set the latency back to zero then things clear up and full 10 Gb is achieved. Iperf performs the same with the latency set to 0 or .5 ms for each direction. We would like to get RoCE to work over high-latency high-bandwidth links. If someone has some ideas on how to resolve this issue, I'd love to hear them.

       

      Commands run on the router server:

      for i in 2 3; do tc qdisc change dev eth${i} root netem delay .5ms; done

       

      # brctl show
      bridge name     bridge id               STP enabled     interfaces
      rleblanc                8000.f452147ce541       no              eth2
                                                              eth3

       

      The iser target is a 100 GB RAM disk exported via iser. I format the disk on the initiator with ext4 and then run this fio command:

      echo "3" > /proc/sys/vm/drop_caches; fio --rw=read --bs=4K --size=1G --numjobs=40 --name=worker.matt --group_reporting

       

      I see these messages on the initiator:

      [25863.623453] 00000000 00000000 00000000 00000000

      [25863.628564] 00000000 00000000 00000000 00000000

      [25863.633634] 00000000 00000000 00000000 00000000

      [25863.638619] 00000000 08007806 250003c7 0b0190d3

      [25863.643593] iser: iser_handle_wc: wr id ffffffffffffffff status 6 vend_err 78

      [25863.651180]  connection40:0: detected conn error (1011)

      [25874.368881] mlx5_warn:mlx5_1:dump_cqe:257:(pid 0): dump error cqe

      [25874.375619] 00000000 00000000 00000000 00000000

      [25874.380690] 00000000 00000000 00000000 00000000

      [25874.385712] 00000000 00000000 00000000 00000000

      [25874.390693] 00000000 08007806 250003c8 0501ddd3

      [25874.395681] iser: iser_handle_wc: wr id ffffffffffffffff status 6 vend_err 78

      [25874.403283]  connection40:0: detected conn error (1011)

      [25923.829903] mlx5_warn:mlx5_1:dump_cqe:257:(pid 0): dump error cqe

      [25923.836663] 00000000 00000000 00000000 00000000

      [25923.841724] 00000000 00000000 00000000 00000000

      [25923.846752] 00000000 00000000 00000000 00000000

      [25923.851733] 00000000 08007806 250003c9 510134d3

      [25923.856709] iser: iser_handle_wc: wr id ffffffffffffffff status 6 vend_err 78

      [25923.864308]  connection40:0: detected conn error (1011)

      [25943.184313] mlx5_warn:mlx5_1:dump_cqe:257:(pid 0): dump error cqe

      [25943.191079] 00000000 00000000 00000000 00000000

      [25943.196208] 00000000 00000000 00000000 00000000

      [25943.201287] 00000000 00000000 00000000 00000000

      [25943.206281] 00000000 08007806 250003ca 1afdbdd3

      [25943.211272] iser: iser_handle_wc: wr id ffffffffffffffff status 6 vend_err 78

      [25943.218901]  connection40:0: detected conn error (1011)

      [25962.538633] mlx5_warn:mlx5_1:dump_cqe:257:(pid 0): dump error cqe

      [25962.545396] 00000000 00000000 00000000 00000000

      [25962.550475] 00000000 00000000 00000000 00000000

      [25962.555551] 00000000 00000000 00000000 00000000

      [25962.560533] 00000000 08007806 250003cb 21012ed3

      [25962.565526] iser: iser_handle_wc: wr id ffffffffffffffff status 6 vend_err 78

      [25962.573155]  connection40:0: detected conn error (1011)

      [25973.291038] mlx5_warn:mlx5_1:dump_cqe:257:(pid 0): dump error cqe

      [25973.297861] 00000000 00000000 00000000 00000000

      [25973.302978] 00000000 00000000 00000000 00000000

      [25973.308025] 00000000 00000000 00000000 00000000

      [25973.313014] 00000000 08007806 250003cc 1901d2d3

      [25973.318004] iser: iser_handle_wc: wr id ffffffffffffffff status 6 vend_err 78

      [25973.325601]  connection40:0: detected conn error (1011)

      [26039.955899] mlx5_warn:mlx5_1:dump_cqe:257:(pid 0): dump error cqe

      [26039.962690] 00000000 00000000 00000000 00000000

      [26039.967825] 00000000 00000000 00000000 00000000

      [26039.972894] 00000000 00000000 00000000 00000000

      [26039.977891] 00000000 08007806 250003cd 850172d3

      [26039.982905] iser: iser_handle_wc: wr id ffffffffffffffff status 6 vend_err 78

      [26039.990512]  connection40:0: detected conn error (1011)

      [26067.411753] mlx5_warn:mlx5_1:dump_cqe:257:(pid 0): dump error cqe

      [26067.418598] 00000000 00000000 00000000 00000000

      [26067.423733] 00000000 00000000 00000000 00000000

      [26067.428832] 00000000 00000000 00000000 00000000

      [26067.433826] 00000000 08007806 250003ce 092977d3

      [26067.438818] iser: iser_handle_wc: wr id ffffffffffffffff status 6 vend_err 78

      [26067.446462]  connection40:0: detected conn error (1011)

       

       

      There are no messages on the target server.