HowTo Configure OVS over RHEL 7 for VXLAN Offload of ConnectX-3 Pro Adapters

Version 14

    This post shows how to configure Open vSwitch (OVS) over RHEL 7 to enable VXLAN offload on ConnectX-3 Pro adapters.





    Basic setup with two servers (Server-A & Server-B) connected with the 40/56GbE and a switch.



    RHEL 7 installation and prerequisite


    1. Install RHEL 7 on the two servers plus MLNX_OFED driver (the MLNX_EN driver is fine as well).


    2. Verify openvswitch configuration on the kernel is as follows:

    # vi /usr/src/kernels/3.10.0-123.e17.x86_64/.config


    Verify that you have the following parameters configured as follows within the file:











    3. Enable flow steering (DMFS) for the mlnx_core kernel module for both servers. The debug level is important for the dmsg logging.

    # cat /etc/modprobe.d/mlx4.conf


    #### enable DMFS --> enable VXLAN offloads on ConnectX3-Pro ####

    options mlx4_core log_num_mgm_entry_size=-1 debug_level=1


    4. Optionally, set the VXLAN UDP port (the default VXLAN port number is 4789 by IANA)

    # cat /etc/modprobe.d/vxlan.conf


    options vxlan udp_port=4789


    5. Verify the UDP tunneling is enabled via ethtool and dmesg.

    # ethtool -k eth2 | grep tx-udp_tnl-segmentation

    tx-udp_tnl-segmentation: on


    # dmesg

    mlx4_core 0000:81:00.0: Device manage flow steering support

    mlx4_core 0000:81:00.0: TCP/IP offloads/flow-steering for VXLAN support mlx4_core 0000:81:00.0: Steering mode is: Device managed flow steering, oper_log_mgm_entry_size = 7, log_num_mgm_entry_size = -1

    mlx4_core 0000:81:00.0: Tunneling offload mode is: vxlan

    mlx4_en: eth2: Setting RSS context tunnel type to RSS on inner headers


    5. Install relevant software packages.

    # yum install gcc make python-devel openssl-devel kernel-devel graphviz kernel-debug-devel autoconf automake rpm-build redhat-rpm-config libtool


    6. Restart the driver

    /etc/init.d/openibd restart


    Installing OVS on both servers


    1. Download OVS from

    # wget


    2. Open the tar file

    # tar -xzvf openvswitch-2.3.1.tar.gz


    3. Compile OVS

    # cd openvswitch-2.3.1

    # ./


    # ./configure


    # make dist

    Once the compilation is done, you should see a new openvswitch-2.3.1.tar.gz on that folder.


    # ls

    acinclude.m4    COPYING            INSTALL.XenServer         PORTING

    aclocal.m4      datapath           IntegrationGuide          python

    AUTHORS         debian             lib                       README

    autom4te.cache  DESIGN             libtool                   README-lisp         FAQ                m4                        REPORTING-BUGS

    build-aux       include            Makefile                  rhel

    BUILD.Windows   INSTALL                 stamp-h1

    CodingStyle     INSTALL.Debian               tests

    config.h        INSTALL.DPDK               third-party     INSTALL.Fedora     NEWS                      TODO    INSTALL.KVM        NOTICE                    tutorial

    config.log      INSTALL.Libvirt    ofproto                   utilities

    config.status   INSTALL.NetBSD     OPENFLOW-1.1+             vswitchd

    configure       INSTALL.RHEL       openvswitch-2.3.1.tar.gz  vtep    INSTALL.SSL        ovsdb                     WHY-OVS

    CONTRIBUTING    INSTALL.userspace  package.m4                xenserver


    Note: It is recommended to review the INSTALL.RHEL readme file as well.


    4. Copy the complied openvswitch-2.3.1.tar.gz file to temporary and SOURCE folder as follows.

    # cp openvswitch-2.3.1.tar.gz /tmp

    # cp openvswitch-2.3.1.tar.gz /root/rpmbuild/SOURCES


    5. Go to temporary folder, extract .tar file and build software.

    # cd /tmp

    # tar -xzvf openvswitch-2.3.1.tar.gz


    # cd openvswitch-2.3.1


    # rpmbuild –bb rhel/openvswitch.spec


    # cp rhel/openvswitch-kmod* /root/rpmbuild/SOURCES

    # rpmbuild –bb rhel/openvswitch-kmod-rhel6.spec


    Note: After building (-bb) make sure you get the return code exit 0.


    6. Install ovs rpms kmod-openvswitch, openvswitch, and openvswitch-debuginfo.

    # rpm –ivh /root/rpmbuild/RPMS/x86_64/kmod-openvswitch-2.3.1-1.e17.x86_64.rpm


    # rpm –ivh /root/rpmbuild/RPMS/x86_64/openvswitch-2.3.1-1.e17.x86_64.rpm


    # rpm –ivh /root/rpmbuild/RPMS/x86_64/openvswitch-debuginfo-2.3.1-1.e17.x86_64.rpm



    7. Verify that ovs modules are installed.

    # find / –name *openvswitch*.ko





    # lsmod | grep openvswitch

    openvswitch            80138  0

    gre                    13808  1 openvswitch

    vxlan                  37584  2 mlx4_en,openvswitch

    libcrc32c              12644  2 xfs,openvswitch


    8. Start the service

    # rhel/etc_init.d_openswitch start

    Starting ovsdb-server [  OK  ]

    Configuring Open vSwitch system IDs [  OK  ]

    Starting ovs-vswitchd [  OK  ]

    Enabling remote OVSDB managers [  OK  ]

    # rhel/etc_init.d_openswitch statue

    ovsdb-server is running with pid 5793

    ovs-vswitchd is running with pid 5803


    9. Verify that OVS is running

    [root@rhel7 script]# ovs-vsctl show


        ovs_version: "2.3.1"


    To configure VXLAN on over OVS, refer to this post

    HowTo Configure VXLAN for ConnectX-3 Pro (Linux OVS)