0 Replies Latest reply on Sep 4, 2018 11:52 PM by lucifer_lee

    testing beegfs with rdma using connectx-3 pro

    lucifer_lee
      I have used vdbench and fio to test for RDMA connections and TCP connections.

       

      But the weird is ,RDMA performence is worse than TCP.

      Hardware and Software Used:

      two machines with no switch and the cards are plugged in back to back

      HCA: Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

      os: CentOS Linux release 7.4.1708 (Core) 3.10.0-693.el7.x86_64

      OFED: MLNX_OFED_LINUX-4.4-2.0.7.0-rhel7.4-x86_64.iso

      the parameter file is:

      messagescan=no
      hd=default,vdbench=/root/download/vdbench50407,user=root,shell=ssh
      hd=hd1,system=stor01
      hd=hd2,system=stor02
      fsd=fsd1,anchor=/mnt/beegfs/largetest2,depth=1,width=5,files=10,size=100m,shared=yes
      fwd=format,threads=16,xfersize=1m
      fwd=default,xfersize=1m,fileio=random,fileselect=random,rdpct=60,threads=16
      fwd=fwd1,fsd=fsd1,host=hd1
      fwd=fwd2,fsd=fsd1,host=hd2
      rd=rd1,fwd=fwd*,fwdrate=max,format=restart,elapsed=600,interval=1

      the result is:

       

       
      largetest 100MRDMATCP
      read(MB/s)13222008
      write(MB/s)883.51342
      total(MB/s)2205.93351.5

       

      anyone has tested beegfs for the difference between RDMA and TCP?