Overview

 

I knew Mellanox OFED 4.2 already contains Soft-RoCE kernel module.

so I tried. Basically, I just followed this instruction.

 

HowTo Configure Soft-RoCE

  • OS: Ubuntu 17.04
  • Mellanox OFED 4.2

 

hostipRoCE interfacenote
host1192.168.100.11enp0s8ibv_rc_pingpong server
host2192.168.100.12enp0s8ibv_rc_pingpong client

 

Step1 Install OFED

 

wget 'http://content.mellanox.com/ofed/MLNX_OFED-4.2-1.2.0.0/MLNX_OFED_LINUX-4.2-1.2.0.0-ubuntu17.04-x86_64.tgz'
tar xvfz ./MLNX_OFED_LINUX-4.2-1.2.0.0-ubuntu17.04-x86_64.tgz
cd MLNX_OFED_LINUX-4.2-1.2.0.0-ubuntu17.04-x86_64
sudo ./mlnxofedinstall -q

 

 

Step2 Start openibd servcie

 

sudo /etc/init.d/openibd start

 

Step3 Get rxe_cfg command.

 

I couldn't find rxe_cfg command in Mellanox OFED package. Instead, I found it in librxe_41mlnx1.orig.tar.gz source codes.

 

cd src
tar xvfz ./MLNX_OFED_SRC-4.2-1.2.0.0.tgz
cd MLNX_OFED_SRC-4.2-1.2.0.0/SOURCES
tar xvfz ./librxe_41mlnx1.orig.tar.gz
cd librxe-41mlnx1

 

Step4 Start Soft-RoCE service.

 

sudo ./rxe_cfg start
  Name    Link  Driver  Speed  NMTU  IPv4_addr  RDEV  RMTU
  enp0s3  yes   e1000
  enp0s8  yes   e1000

 

 

Step4 Add Soft-RoCE interface.

 

sudo ./rxe_cfg add enp0s8
sudo ./rxe_cfg status
  Name    Link  Driver  Speed  NMTU  IPv4_addr  RDEV  RMTU
  enp0s3  yes   e1000
  enp0s8  yes   e1000                           rxe0  1024  (3)

 

 

Step5 Execute ibv_rc_pingpong.

 

host1

 

ibv_rc_pingpong -d rxe0 -g 0
  local address:  LID 0x0000, QPN 0x000011, PSN 0x7c660e, GID fe80::a00:27ff:fe64:bbf1
  remote address: LID 0x0000, QPN 0x000011, PSN 0x34e576, GID fe80::a00:27ff:febf:ca7d
8192000 bytes in 0.87 seconds = 75.68 Mbit/sec
1000 iters in 0.87 seconds = 865.98 usec/iter

 

host2

 

ibv_rc_pingpong -d rxe0 -g 0 192.168.100.11
  local address:  LID 0x0000, QPN 0x000011, PSN 0x34e576, GID fe80::a00:27ff:febf:ca7d
  remote address: LID 0x0000, QPN 0x000011, PSN 0x7c660e, GID fe80::a00:27ff:fe64:bbf1
8192000 bytes in 0.86 seconds = 76.20 Mbit/sec
1000 iters in 0.86 seconds = 860.03 usec/iter

 

Happy Soft-RoCE hacking!!.