Getting started with VXLAN over ConnectX-3 Pro adapters on CentOS 7.1

Version 12

    This post shows basic (minimum) VXLAN setup using ConnectX-3 Pro adapters over CentOS 7.1 (kerenl 3.10.0-229) Inbox driver via Mellanox SX1710 Ethernet switch.

     

     

    References

     

    Setup

    Two servers equipped with ConnectX-3 Pro adapter, installed with CentOS 7.1 Inbox driver and connected via a switch (or even back to back).

     

    vxlan basic - New Page (1).png

     

    Configuration

    Basic VXLAN Configuration (No CPU Offload)

    The following configuration is an example aligned with the setup. VXLAN offload is not configured.

     

    1. Set the IP addressed of the physical enp2s0 interface. Verify that there is a ping between the two servers.

     

    For Server S3:

    # ifconfig enp2s0 11.11.11.3/24 up

     

    For Server S4:

    # ifconfig enp2s0 11.11.11.4/24 up

     

    2. Set the VXLAN port number in /etc/modprobe.d/vxlan.conf

    # cat /etc/modprobe.d/vxlan.conf

     

    #### Set the VXLAN UDP port ####

    options vxlan udp_port=4789

     

    Note: If the UDP port number is not changed, the Linux OS uses a default non standard UDP port number.

     

    3. Create vxlan0 interface with group id 42 and multicast group 239.1.1.1 over the enp2s0 interface (on both servers).

    # ip link add vxlan0 type vxlan id 42 group 239.1.1.1 dev enp2s0

    vxlan: destination port not specified

    Will use Linux kernel default (non-standard value)

    Use 'dstport 4789' to get the IANA assigned value

    Use 'dstport 0' to get default and quiet this message

     

    4. Set the IP address for the vxlan0 interface and enable the interface.

     

    For Server S3:

    # ifconfig vxlan0 12.12.12.3/24 up

     

    For Server S4:

    # ifconfig vxlan0 12.12.12.4/24 up

     

    6. Check the configuration:

    This is an example from Server S3:

     

    # ifconfig enp2s0

    enp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500

            inet 11.11.11.3  netmask 255.255.255.0  broadcast 11.11.11.255

            inet6 fe80::e61d:2dff:fe26:3ce1  prefixlen 64  scopeid 0x20<link>

            ether e4:1d:2d:26:3c:e1  txqueuelen 1000  (Ethernet)

            RX packets 131  bytes 20918 (20.4 KiB)

            RX errors 0  dropped 0  overruns 0  frame 0

            TX packets 122  bytes 19565 (19.1 KiB)

            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

     

    # ifconfig vxlan0

    vxlan0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450

            inet 12.12.12.3  netmask 255.255.255.0  broadcast 0.0.0.0

            inet6 fe80::b81b:81ff:fead:5211  prefixlen 64  scopeid 0x20<link>

            ether ba:1b:81:ad:52:11  txqueuelen 0  (Ethernet)

            RX packets 55  bytes 4418 (4.3 KiB)

            RX errors 0  dropped 0  overruns 0  frame 0

            TX packets 78  bytes 9930 (9.6 KiB)

            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

     

    # ip -d link show vxlan0

    8: vxlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT

        link/ether ba:1b:81:ad:52:11 brd ff:ff:ff:ff:ff:ff promiscuity 0

        vxlan id 42 group 239.1.1.1 dev enp2s0 srcport 32768 61000 dstport 4789 ageing 300

     

    7. Verify that ping between the servers over the vxlan0 interface (12.12.12.3, 12.12.12.4)

     

    8. Run wireshark on one of the servers to capture the frames:

     

    • Capture Example of the enp2s0 interface:

              You can see that this is a UDP packet sent to the 11.11.11.0 network (the VXLAN frame is embedded in the payload).

    1.PNG

     

    • Capture Example of the vxlan0 interface:

              You can see the ICMP packet.

    2.PNG

     

    9. Check the ARP and routing table on the servers.

    # arp -n

    Address                  HWtype  HWaddress           Flags Mask            Iface

    11.11.11.3               ether   e4:1d:2d:26:3c:e1   C                     enp2s0

    12.12.12.3               ether   ba:1b:81:ad:52:11   C                     vxlan0

    ....

     

    # route

    Kernel IP routing table

    Destination     Gateway         Genmask         Flags Metric Ref    Use Iface

    ...

    11.11.11.0      0.0.0.0         255.255.255.0   U     0      0        0 enp2s0

    12.12.12.0      0.0.0.0         255.255.255.0   U     0      0        0 vxlan0

    [root@mti-mar-s4 ~]#

     

     

    10. The handshake can be viewed with wireshark using the attached vxlan.pcapng file (scroll down).

     

    VXLAN Configuration (CPU Offload)

    Once you understand and manage to enable the VXLAN interface, you are ready to add CPU offload support.

    To learn more about the CPU offload, refer to VXLAN Considerations for ConnectX-3 Pro.

     

    1. Check the current CPU offload status:

    # ethtool -k enp2s0 | grep tnl

    tx-udp_tnl-segmentation: off [fixed]

    As you can see, the offload (tx-udp-tnl-segmentation) is Off.

     

    2. In order to enable VXLAN offload, enable the Device-Managed Flow Steering (DMFS) on the mlx4_core module parameters. To do so, create/edit the /etc/modprobe.d/mlx4_core.conf file as follows:

    options mlx4_core log_num_mgm_entry_size=-1 debug_level=1

     

    3. Reload the kernel modules and reconfigure the interfaces.

    Here is an example from Server S3:

    # rmmod mlx4_en

    # rmmod mlx4_ib

    # rmmod mlx4_core

    # rmmod vxlan

    # modprobe vxlan

    # modprobe mlx4_en

    # modprobe mlx4_core

    # modprobe mlx4_ib

    # ifconfig enp2s0 11.11.11.3/24 up

    # ip link add vxlan0 type vxlan id 42 group 239.1.1.1 dev enp2s0

    # ifconfig vxlan0 12.12.12.3/24 up

     

    4. Check the offload status.

    # ethtool -k enp2s0 | grep tnl

    tx-udp_tnl-segmentation: on [requested off]

     

    Note: The "ip link add vxlan0 ... " command is required to be executed in order to see that the offload is actually enabled (=on).

     

    5. Compare the performance using any performance tool (e.g. iperf)

     

    Troubleshooting

    1. If you change the VXLAN parameters after creating the interface (for example UDP port), you will need to remove the VXLAN/mlx4_en/mlx4_core modules from the kernel and re-add it.

    # rmmod mlx4_en

    # rmmod mlx4_ib

    # rmmod mlx4_core

    # rmmod vxlan

    # modprobe vxlan

    # modprobe mlx4_e

    # modprobe mlx4_en

    # modprobe mlx4_core

    # modprobe mlx4_ib