HowTo Configure OVS over RHEL 7 for VXLAN Offload of ConnectX-3 Pro Adapters

Version 14

    This post shows how to configure Open vSwitch (OVS) over RHEL 7 to enable VXLAN offload on ConnectX-3 Pro adapters.

     

    References

     

    Setup

    Basic setup with two servers (Server-A & Server-B) connected with the 40/56GbE and a switch.

     

    Configuration

    RHEL 7 installation and prerequisite

     

    1. Install RHEL 7 on the two servers plus MLNX_OFED driver (the MLNX_EN driver is fine as well).

     

    2. Verify openvswitch configuration on the kernel is as follows:

    # vi /usr/src/kernels/3.10.0-123.e17.x86_64/.config

     

    Verify that you have the following parameters configured as follows within the file:

    CONFIG_OPENVSWITCH=m

    CONFIG_OPENVSWITCH_VXLAN=y

    CONFIG_VXLAN=m

    CONFIG_NET_VENDOR_MELLANOX=y

    CONFIG_MLX4_EN=m

    CONFIG_MLX4_EN_DCB=y

    CONFIG_MLX4_CORE=m

    CONFIG_MLX4_DEBUG=y

    CONFIG_MLX4_INFINIBAND=m

     

    3. Enable flow steering (DMFS) for the mlnx_core kernel module for both servers. The debug level is important for the dmsg logging.

    # cat /etc/modprobe.d/mlx4.conf

     

    #### enable DMFS --> enable VXLAN offloads on ConnectX3-Pro ####

    options mlx4_core log_num_mgm_entry_size=-1 debug_level=1

     

    4. Optionally, set the VXLAN UDP port (the default VXLAN port number is 4789 by IANA)

    # cat /etc/modprobe.d/vxlan.conf

     

    options vxlan udp_port=4789

     

    5. Verify the UDP tunneling is enabled via ethtool and dmesg.

    # ethtool -k eth2 | grep tx-udp_tnl-segmentation

    tx-udp_tnl-segmentation: on

     

    # dmesg

    mlx4_core 0000:81:00.0: Device manage flow steering support

    mlx4_core 0000:81:00.0: TCP/IP offloads/flow-steering for VXLAN support mlx4_core 0000:81:00.0: Steering mode is: Device managed flow steering, oper_log_mgm_entry_size = 7, log_num_mgm_entry_size = -1

    mlx4_core 0000:81:00.0: Tunneling offload mode is: vxlan

    mlx4_en: eth2: Setting RSS context tunnel type to RSS on inner headers

     

    5. Install relevant software packages.

    # yum install gcc make python-devel openssl-devel kernel-devel graphviz kernel-debug-devel autoconf automake rpm-build redhat-rpm-config libtool

     

    6. Restart the driver

    /etc/init.d/openibd restart

     

    Installing OVS on both servers

     

    1. Download OVS from openvswitch.org

    # wget http://openvswitch.org/releases/openvswitch-2.3.1.tar.gz

     

    2. Open the tar file

    # tar -xzvf openvswitch-2.3.1.tar.gz

     

    3. Compile OVS

    # cd openvswitch-2.3.1

    # ./boot.sh

    ...

    # ./configure

    ...

    # make dist

    Once the compilation is done, you should see a new openvswitch-2.3.1.tar.gz on that folder.

     

    # ls

    acinclude.m4    COPYING            INSTALL.XenServer         PORTING

    aclocal.m4      datapath           IntegrationGuide          python

    AUTHORS         debian             lib                       README

    autom4te.cache  DESIGN             libtool                   README-lisp

    boot.sh         FAQ                m4                        REPORTING-BUGS

    build-aux       include            Makefile                  rhel

    BUILD.Windows   INSTALL            Makefile.am               stamp-h1

    CodingStyle     INSTALL.Debian     Makefile.in               tests

    config.h        INSTALL.DPDK       manpages.mk               third-party

    config.h.in     INSTALL.Fedora     NEWS                      TODO

    config.h.in~    INSTALL.KVM        NOTICE                    tutorial

    config.log      INSTALL.Libvirt    ofproto                   utilities

    config.status   INSTALL.NetBSD     OPENFLOW-1.1+             vswitchd

    configure       INSTALL.RHEL       openvswitch-2.3.1.tar.gz  vtep

    configure.ac    INSTALL.SSL        ovsdb                     WHY-OVS

    CONTRIBUTING    INSTALL.userspace  package.m4                xenserver

     

    Note: It is recommended to review the INSTALL.RHEL readme file as well.

     

    4. Copy the complied openvswitch-2.3.1.tar.gz file to temporary and SOURCE folder as follows.

    # cp openvswitch-2.3.1.tar.gz /tmp

    # cp openvswitch-2.3.1.tar.gz /root/rpmbuild/SOURCES

     

    5. Go to temporary folder, extract .tar file and build software.

    # cd /tmp

    # tar -xzvf openvswitch-2.3.1.tar.gz

    ...

    # cd openvswitch-2.3.1

    ..

    # rpmbuild –bb rhel/openvswitch.spec

     

    # cp rhel/openvswitch-kmod* /root/rpmbuild/SOURCES

    # rpmbuild –bb rhel/openvswitch-kmod-rhel6.spec

    ...

    Note: After building (-bb) make sure you get the return code exit 0.

     

    6. Install ovs rpms kmod-openvswitch, openvswitch, and openvswitch-debuginfo.

    # rpm –ivh /root/rpmbuild/RPMS/x86_64/kmod-openvswitch-2.3.1-1.e17.x86_64.rpm

    ...

    # rpm –ivh /root/rpmbuild/RPMS/x86_64/openvswitch-2.3.1-1.e17.x86_64.rpm

    ...

    # rpm –ivh /root/rpmbuild/RPMS/x86_64/openvswitch-debuginfo-2.3.1-1.e17.x86_64.rpm

    ...

     

    7. Verify that ovs modules are installed.

    # find / –name *openvswitch*.ko

    /root/rpmbuild/BUILD/openvswitch-2.3.1/_default/datapath/linux/openvswitch.ko

    /usr/lib/modules/3.10.0-123.el7.x86_64/extra/openvswitch/openvswitch.ko

    /usr/lib/modules/3.10.0-123.el7.x86_64/kernel/net/openvswitch/openvswitch.ko

     

    # lsmod | grep openvswitch

    openvswitch            80138  0

    gre                    13808  1 openvswitch

    vxlan                  37584  2 mlx4_en,openvswitch

    libcrc32c              12644  2 xfs,openvswitch

     

    8. Start the service

    # rhel/etc_init.d_openswitch start

    Starting ovsdb-server [  OK  ]

    Configuring Open vSwitch system IDs [  OK  ]

    Starting ovs-vswitchd [  OK  ]

    Enabling remote OVSDB managers [  OK  ]

    # rhel/etc_init.d_openswitch statue

    ovsdb-server is running with pid 5793

    ovs-vswitchd is running with pid 5803

     

    9. Verify that OVS is running

    [root@rhel7 script]# ovs-vsctl show

    184206cf-a569-4aa8-8b74-08a36980e431

        ovs_version: "2.3.1"

     

    To configure VXLAN on over OVS, refer to this post

    HowTo Configure VXLAN for ConnectX-3 Pro (Linux OVS)