Mirantis Fuel 8.0 Support for Mellanox ConnectX-3 Pro Adapters

Version 10

    This post shows the major updates to the Mirantis Fuel 8.0 and Mellanox ConnectX-3 adapters.




    The major updates include:

    • Support for Fuel and OpenStack Liberty
    • UI changes
    • Bootstrap VPI creation on Fuel Master
    • Direct port usage for Ethernet and InfiniBand (IB)
    • VXLAN offloading in Neutron with tunneling
    • An OFED driver version upgrade to 3.1-1.5.3.
    • Neutron with VLAN and QoS
    • NEO auto-provisioning support
    • Release notes


    Definitions, Acronyms and Abbreviations



    SR-IOV stands for "Single Root I/O Virtualization". It is a specification that allows a PCI device to appear virtually in multiple virtual machines (VMs), each of which has its own virtual function. The specification defines virtual functions (VFs) for the VMs and a physical function for the hypervisor. Using SR-IOV in a cloud infrastructure helps reaching higher performance since traffic bypasses the TCP/IP stack in the kernel


    iSER stands for "iSCSI Extensions for RDMA". It is an extension of the data transfer model of iSCSI, a storage networking standard for TCP/IP. iSER enables the iSCSI protocol to take advantage of the RDMA protocol suite to supply higher bandwidth for block storage transfers (zero time copy behavior). To that fact, it eliminates the TCP/IP processing overhead while preserving the compatibility with iSCSI protocol.


    RDMA stands for "Remote Direct Memory Access". It is a technology that enables to read and write data from remote server without involving the CPU. It reduces latency and increases throughput. In addition, the CPU is free to perform other tasks.

    ConnectX-3 Pro

    ConnectX-3 Pro adapter cards with Virtual Protocol Interconnect (VPI) supporting InfiniBand and Ethernet connectivity provide the highest performing and most flexible interconnect solution for PCI Express Gen3 servers used in Enterprise Data Centers, High-Performance Computing, and Embedded environments.


    A computer-networking communications standard used in high-performance computing, features very high throughput and very low latency. It is used for data interconnect both among and within computers. InfiniBand is also utilized as either a direct, or switched interconnect between servers and storage systems, as well as an interconnect between storage systems.

    VXLAN Offload

    Virtual Extensible LAN (VXLAN) is a network virtualization technology that attempts to improve the scalability problems associated with large cloud computing deployments.


    Quality of Service (QoS) is defined as the ability to guarantee certain network requirements like bandwidth, latency, jitter and reliability in order to satisfy a Service Level Agreement (SLA) between an application provider and end users.



    Mellanox Plugin


    The Mellanox Fuel plugin is a bundle of scripts, packages and metadata that will extend Fuel and add Mellanox features, such as SR-IOV for networking and the iSER protocol for storage.

    Beginning with Fuel version 5.1, the application supported configuration of Mellanox ConnectX-3 Pro network adapters. The end result was the accelerated performance of compute and storage traffic.

    This plugin implements the following performance enhancements:

    • Compute nodes network enhancements include:
      • SR-IOV based networking
      • QoS for VM traffic
      • VXLAN traffic offload
    • Cinder nodes use iSER block storage as the iSCSI transport, rather than the default iSCSI over TCP.

    These features reduce CPU overhead, boost throughput, reduce latency, and enable network traffic to bypass the software switch layer (for example: Open vSwitch).

    The Mellanox Plugin integration with the Mellanox NEO SDN Controller enables switch VLAN auto provisioning and port configuration for Ethernet and SM partition key (PK) auto provisioning for InfiniBand networks.



    The Mellanox ConnectX-3 Pro adapter family supports up to 40/56 Gb. To reach a 56 Gb speed in your network with ConnectX-3 Pro adapters, you must use Mellanox Ethernet / InfiniBand switches supporting 56 Gb (e.g. SX1710, SX6710). The switch ports should be configured specifically so they can use the 56 Gb speed. No additional configuration is required on the adapter side. For additional information about how to run in 56GbE speed, see HowTo Configure 56GbE Link on Mellanox Adapters and Switches.

    For detailed setup and hardware and software requirements please see How to configure Fuel Ethernet cloud or How to configure Fuel InfiniBand cloud.



    • Mellanox SR-IOV and iSER are supported only when choosing Neutron with VLAN segmentation.
    • OVS bonding and Mellanox SR-IOV based networking over the Mellanox ConnectX-3 Pro adapter family are not supported.
    • In order to use the SR-IOV feature, one should choose KVM hypervisor and “Neutron with VLAN segmentation” in the Network settings tab
    • ConnectX-3 Pro adapters must be used in order to enable VXLAN hardware offload over Ethernet networks.
    • The QoS feature is implemented only for Ethernet networks using ConnectX-3 Pro adapters.
    • InfiniBand is configured by using OpenSM only.
    • VM Live and non-Live Migration does not work when SR-IOV is enabled.
    • When using iSER and bonding, the storage interface will be a virtual function under the first port, and will not support failover

    • When using SR-IOV and bonding, the virtual functions will be created and allocated from the first interface, and failover is not supported


    Installation Guide


    Mellanox Plugin Backend Configuration

    If you plan to enable VM-to-VM RDMA and to use iSER storage transport, you need to configure the switching fabric so it supports the features described below.

    Ethernet Network

    • Configure the required VLANs and enable flow control on the Ethernet switch ports.

    • All related VLANs should be enabled on the 40/56GbE switch (Private, Management, and Storage networks).

    • For Mellanox switches you can refer to the Mirantis planning guide for an example of how to configure the switch for Ethernet deployment.


    InfiniBand Network

    If you use OpenSM, you need to enable virtualization and allow all PKs:

    1. Create a new opensm.conf file.

    # opensm -c /etc/opensm/opensm.conf

      2. Enable virtualization by editing /etc/opensm/opensm.conf and changing the allow_both_pkeys value to TRUE.

    allow_both_pkeys TRUE

      3. Define the partition keys, which are analog for Ethernet VLAN. Each VLAN will be mapped to one PK. Add/Change the following with the command #vi /etc/opensm/partitions.conf file as the example shows:

    management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
    vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
    vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
    vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
    . . .
    vlan100=0x64, ipoib, sl=0, defmember=full : ALL;


    4. Restart OpenSM.

    # /etc/init.d/opensmd restart


    Mellanox Plugin Installation

    To install the Mellanox plugin, follow these steps:

    1. Install Fuel Master node. For more information on how to create a Fuel Master node, please see Mirantis Fuel 8.0 documentation.


    2. Download the plugin rpm file for MOS8.0 from Fuel Plugin Catalog.


    3. To copy the plugin on the already-installed Fuel Master node, use the scp command as follows:

    # scp mellanox-plugin-3.0-3.0.0-1.noarch.rpm root@<Fuel_Master_ip>:/tmp


    4. Install the plugin:

    # cd /tmp
    # fuel plugins --install mellanox-plugin-3.0-3.0.0-1.noarch.rpm


    5. Verify that the plugin was installed successfully by having it listed using fuel plugins command:

    # fuel plugins
    id | name              | version | package_version---|-------------------|---------|---------------- 1 | mellanox-plugin   | 3.0.0   | 3.0.0


    6. Create new bootstrap image by running create_mellanox_vpi_bootstrap can be used.

    [root@fuel ~]# create_mellanox_vpi_bootstrap can be used

    Try to build image with data:


    certs: null

    container: {format: tar.gz, meta_file: metadata.yaml}

    . . .

    . . .

    . . .

    Bootstrap image f790e9f8-5bc5-4e61-9935-0640f2eed949 has been activated.

    7. Boot your target nodes with the new bootstrap image you just created.

    If you have already discovered nodes, you can either reboot them manually or use the reboot_bootstrap_nodes command. Run reboot_bootstrap_nodes –h to access help.


    8. Create an environment. For more information, see how to create an environment.

    Both of the main network configurations are supported:

    • Neutron with VLAN segmentation

    • Neutron with tunneling segmentation