HowTo Configure VXLAN for ConnectX-3 Pro (Linux Bridge)

Version 6

    This post explains how to configure VXLAN on Linux used on ConnectX®-3 Pro adapter cards via Linux Bridge.





    Server requirements

    • ConnectX-3 Pro
    • Operating system and kernel options:
      • upstream Linux 3.14 or later
      • RHEL6.5 (kernel 2.6.32-431.el6) or late
      • KVM Hypervisor using para-virtual NIC (e.g. virtio with vhost backend on the hypervisor)


    • MLNX_OFED (2.2 or later) installation is optional, as driver support is inbox (RHEL 7 or Ubuntu 14.04)



    • Two servers connected to Mellanox VMS, each on different VLAN
    • Mellanox VMS switches are configured as routers






    • Both Servers are installed with RHEL6.5 (kernel 2.6.32-431.el6) or late
    • Enable intel_iommu=on and iommu=pt on the kernel grub file for better performance see here for example.



    Server Configuration


    1. Create Linux Bridge on both servers, and enable it.

    # brctl addbr br-vx

    # ip link set br-vx up



    2. Create VXLAN (virtual) interface, using multicast address enable it and add it to the Linux bridge.

    # ip link add vxlan10 type vxlan id 10 group ttl 10 dev eth2

    # ip link set vxlan10 up

    # brctl addif br-vx vxlan10



    3. Create VETH (virtual) interface for the VM connectivity.

    # ip link add type veth

    Note, at this point there are two interfaces added veth0 and veth1 (see the figure above)

    The veth0 is suppose to be connected to the VM, while the veth1 is connected to the Linux bridge.


    4. Set veth0 parameters (IP and MTU)

    Note, the MTU is reduced to 1450 as the VXLAN the and other headers adds 50 bytes.


    Host A configuration:

    # ifconfig veth0 up

    # ifconfig veth0 mtu 1450



    Host B configuration (different IP address):

    # ifconfig veth0 up

    # ifconfig veth0 mtu 1450




    5. Set veth1 parameters (MTU)

    Note, the MTU is reduced to 1450 as the VXLAN the and other headers adds 50 bytes.

    # ifconfig veth1 up

    # ifconfig veth1 mtu 1450



    6. Add the veth1 interface to the Linux bridge.

    # brctl addif br-vx veth1



    7. Verify that the multicast address is used on the interface


    # ip maddr show dev eth2

    9:      eth2

            link  01:00:5e:0a:0a:0a

            link  01:00:5e:00:00:52

            link  33:33:ff:ef:f5:21

            link  33:33:00:00:00:01

            link  01:00:5e:00:00:01



            inet6 ff02::1:ffef:f521

            inet6 ff02::1



    8. Show the Linux bridge configuration:

    # brctl show br-vx

    bridge name     bridge id               STP enabled     interfaces

    br-vx           8000.6a14f8a45731       no              veth1



    9. Verify that VXLAN connectivity is available over the network :


    PING ( 56(84) bytes of data.

    64 bytes from icmp_seq=1 ttl=64 time=0.059 ms



    10. Verify that the multicast group is listed on the switches.

    switch (config) # show ip igmp group

    IGMP Connected Group Membership

    Type: S - Static, D - Dynamic

    Group Address    Type    Interface        Uptime       Expires              Last Reporter       D     Vlan60           00:52:01     00:04:05   

    switch (config) #