Understanding VXLAN Hardware Stateless Offload ethtool Parameters for ConnectX-4

Version 3

    This post discusses the ethtool offload parameters related to VXLAN on the ConnectX-4 adapter.






    The offloads discussed here are enabled by default and recommended to be used with kernel version that includes VXLAN support. The earlier kernels include some of the offloads that benefit VXLAN processing; however, to take advantage of all offload options available by ConnectX-4, it is recommended to use the upstream kernel. In order to understand the offloads, make sure that you understand the VXLAN packet format, refer to VXLAN Considerations for ConnectX-3 Pro for more info.


    Note: There is no ability to fully disable VXLAN offload on ConnectX-4.


    The following ethtool options are configurable using the -K flag. The availability of these options is dependent on kernel and ethtool versions.


    1. Disable/Enable Checksum offload on Rx.

    When enabled it reports “csum complete” on all the packet (inner/outer headers and payload). This means that VXLAN performance on RX csum calculation can be boosted if used on kernels that support checksum complete on RX.

    # ethtool -K <interface> rx <off|on>


    2.  Enable/Disable TSO (TCP Segmentation Offload) for tunneling protocols. This means that all packets will be getting to the NIC already MTU sized, instead of a large packet being segmented by hardware. This might be the most important of TCO offloads for VXLAN packets: it offloads the segmentation of large TX packets (packet size >> MTU) to NIC hardware. For example, in case the inner packet will be 9000 bytes, while the port MTU is 1500 bytes. the segmentation of the packet will be in the HW.

    # ethtool -K <interface> tx-udp_tnl-segmentation <off|on>


    3.  Enable/Disable inner packet checksums offload for for tunneling protocols on the Tx side.

    Note: This is relevant for upsteam kernel 4.6 and above (see http://lists.openwall.net/netdev/2016/03/19/2 ). (As such, and for example, this feature is not supported on RHEL 7.1.)

    ethtool -K eth2  tx-udp_tnl-csum-segmentation <off|on>


    To show the current configuration run:

    # ethtool -k <interface>


    For example:

    # ethtool -k enp2s0f0

    Features for enp2s0f0:

    rx-checksumming: on

    tx-checksumming: on

      tx-checksum-ipv4: on

      tx-checksum-ip-generic: off [fixed]

      tx-checksum-ipv6: on

      tx-checksum-fcoe-crc: off [fixed]

      tx-checksum-sctp: off [fixed]

    scatter-gather: on

      tx-scatter-gather: on

      tx-scatter-gather-fraglist: off [fixed]

    tcp-segmentation-offload: on

      tx-tcp-segmentation: on

      tx-tcp-ecn-segmentation: off [fixed]

      tx-tcp-mangleid-segmentation: off

      tx-tcp6-segmentation: on

    udp-fragmentation-offload: off [fixed]

    generic-segmentation-offload: on

    generic-receive-offload: on

    large-receive-offload: off

    rx-vlan-offload: on

    tx-vlan-offload: on

    ntuple-filters: off [fixed]

    receive-hashing: on

    highdma: on [fixed]

    rx-vlan-filter: on

    vlan-challenged: off [fixed]

    tx-lockless: off [fixed]

    netns-local: off [fixed]

    tx-gso-robust: off [fixed]

    tx-fcoe-segmentation: off [fixed]

    tx-gre-segmentation: off [fixed]

    tx-gre-csum-segmentation: off [fixed]

    tx-ipxip4-segmentation: off [fixed]

    tx-ipxip6-segmentation: off [fixed]

    tx-udp_tnl-segmentation: on

    tx-udp_tnl-csum-segmentation: on

    tx-gso-partial: on

    fcoe-mtu: off [fixed]

    tx-nocache-copy: off

    loopback: off [fixed]

    rx-fcs: off [fixed]

    rx-all: off [fixed]

    tx-vlan-stag-hw-insert: off [fixed]

    rx-vlan-stag-hw-parse: off [fixed]

    rx-vlan-stag-filter: off [fixed]

    l2-fwd-offload: off [fixed]

    busy-poll: off [fixed]

    hw-tc-offload: off [fixed]