HowTo Configure VXLAN for ConnectX-3 Pro (Linux Bridge)

Version 5

    This post explains how to configure VXLAN on Linux used on ConnectX®-3 Pro adapter cards via Linux Bridge.

     

    References

     

     

    Server requirements

    • ConnectX-3 Pro
    • Operating system and kernel options:
      • upstream Linux 3.14 or later
      • RHEL6.5 (kernel 2.6.32-431.el6) or late
      • KVM Hypervisor using para-virtual NIC (e.g. virtio with vhost backend on the hypervisor)

     

    • MLNX_OFED (2.2 or later) installation is optional, as driver support is inbox (RHEL 7 or Ubuntu 14.04)

     

    Setup

    • Two servers connected to Mellanox VMS, each on different VLAN
    • Mellanox VMS switches are configured as routers

     

     

    1.png

     

    Prerequisites

     

     

    Server Configuration

     

    1. Create Linux Bridge on both servers, and enable it.

    # brctl addbr br-vx

    # ip link set br-vx up

       

     

    2. Create VXLAN (virtual) interface, using multicast address enable it and add it to the Linux bridge.

    # ip link add vxlan10 type vxlan id 10 group 224.10.10.10 ttl 10 dev eth2

    # ip link set vxlan10 up

    # brctl addif br-vx vxlan10

      

     

    3. Create VETH (virtual) interface for the VM connectivity.

    # ip link add type veth

    Note, at this point there are two interfaces added veth0 and veth1 (see the figure above)

    The veth0 is suppose to be connected to the VM, while the veth1 is connected to the Linux bridge.

     

    4. Set veth0 parameters (IP and MTU)

    Note, the MTU is reduced to 1450 as the VXLAN the and other headers adds 50 bytes.

     

    Host A configuration:

    # ifconfig veth0 192.168.10.1/24 up

    # ifconfig veth0 mtu 1450

      

     

    Host B configuration (different IP address):

    # ifconfig veth0 192.168.10.2/24 up

    # ifconfig veth0 mtu 1450

      

     

     

    5. Set veth1 parameters (MTU)

    Note, the MTU is reduced to 1450 as the VXLAN the and other headers adds 50 bytes.

    # ifconfig veth1 up

    # ifconfig veth1 mtu 1450

      

     

    6. Add the veth1 interface to the Linux bridge.

    # brctl addif br-vx veth1

      

     

    7. Verify that the multicast address is used on the interface

     

    # ip maddr show dev eth2

    9:      eth2

            link  01:00:5e:0a:0a:0a

            link  01:00:5e:00:00:52

            link  33:33:ff:ef:f5:21

            link  33:33:00:00:00:01

            link  01:00:5e:00:00:01

            inet  224.10.10.10

            inet  224.0.0.1

            inet6 ff02::1:ffef:f521

            inet6 ff02::1

      

     

    8. Show the Linux bridge configuration:

    # brctl show br-vx

    bridge name     bridge id               STP enabled     interfaces

    br-vx           8000.6a14f8a45731       no              veth1

                                                            vxlan10

      

    9. Verify that VXLAN connectivity is available over the network :

    #ping 192.168.10.2

    PING 192.168.10.2 (192.168.10.2) 56(84) bytes of data.

    64 bytes from 192.168.10.2: icmp_seq=1 ttl=64 time=0.059 ms

      

     

    10. Verify that the multicast group is listed on the switches.

    switch (config) # show ip igmp group

    IGMP Connected Group Membership

    Type: S - Static, D - Dynamic

    Group Address    Type    Interface        Uptime       Expires              Last Reporter

    224.10.10.10       D     Vlan60           00:52:01     00:04:05             11.11.6.1

    switch (config) #

     

     

    Other switch verification options can be checked in HowTo Configure IP Multicast (PIM, IGMP) on Mellanox Ethernet Switches.