HowTo Install Mirantis Fuel 5.1/5.1.1 OpenStack with Mellanox Adapters Support (InfiniBand Network)

Version 19
    This post shows how to set up and configure Mirantis Fuel ver. 5.1/5.1.1 (OpenStack Icehouse based on CentOS 6.5) to support Mellanox ConnectX-3 adapters to work in SR-IOV mode for the VMs on the compute nodes, and in iSER (iSCSI over RDMA) transport mode for the storage nodes.
    This post is very similar to HowTo Install Mirantis Fuel 5.1/5.1.1 OpenStack with Mellanox Adapters Support (Ethernet Network), but uses InfiniBand network and not Ethernet network.


    Related references

    Before reading this post, Make sure you are familiar with Mirantis Fuel 5.1/5.1.1 installation procedures.


    Note: Contact Mellanox Support for the Fuel ISO Image.


    Setup Diagram

    5.1 arch IB.png


    Note: Besides the Fuel Master node, all nodes should be connected to all five networks.


    Note: Server’s IPMI and the switches management interfaces wiring and configuration are out of scope.

    You need to ensure that there is management access (SSH) to Mellanox InfiniBand switch SX6036 to perform the configuration.



    Setup BOM


    Fuel Master server1

    DELL PowerEdge R620

    • CPU: 2 x E5-2650 @ 2.00GHz
    • MEM: 128 GB
    • HD: 2 x 900GB SAS 10k in RAID-1

    Cloud Controllers and Compute servers

    • 3 x Controllers
    • 3 x Computes

    DELL PowerEdge R620

    • CPU: 2 x E5-2650 @ 2.00GHz
    • MEM: 128 GB
    • HD: 2 x 900GB SAS 10k in RAID-1
    • NIC: Mellanox ConnectX-3Pro VPI (MCX353-FCCT)
    Cloud Storage server1

    Supermicro X9DR3-F

    • CPU: 2 x E5-2650 @ 2.00GHz
    • MEM: 128 GB
    • HD: 24 x 6Gb/s SATA Intel SSD DC S3500 Series 480GB (SSDSC2BB480G4)
    • RAID Ctrl: LSI Logic MegaRAID SAS 2208 with battery
    • NIC: Mellanox ConnectX-3Pro VPI (MCX353-FCCT)
    Admin  (PXE) and Public switch11Gb switch with VLANs configured to support both networks
    InfiniBand Switch1Mellanox SX6036 FDR 36 port InfiniBand

    16 x 1Gb CAT-6e for Admin (PXE) and Public networks

    7 x 56GbE copper cables up to 2m (MC2207130-XXX)

    Note: You can use Mellanox ConnectX-3 PRO EN (MCX313A-BCCT) or Mellanox ConnectX-3 Pro VPI (MCX353-FCCT) adapter cards.


    Storage server RAID Setup

    • 2 SSD drives in bays 0-1 configured in RAID-1 (Mirror): The OS will be installed on it.
    • 22 SSD drives in bays 3-24 configured in RAID-10: The Cinder volume will be configured on the RAID drive.

    Network Physical Setup

    1. Connect all nodes to the Admin (PXE) 1GbE switch (preferably through the eth0 interface on board).
      It is recommended to write the MAC address of the Controller and Storage servers to make Cloud installation easier (see Controller Node section below in Nodes tab).
    2. Connect all nodes to the Public 1GbE switch (preferably through the eth1 interface on board).
    3. Connect port #1 (ib0) of ConnectX-3 Pro to SX6036 InfiniBand switch (Private, Management, Storage networks).
    Note: The interface names (eth0, eth1, p2p1, etc.) may vary between servers from different vendors.
    Note: Port bonding is not supported when using SR-IOV over the ConnectX-3 adapter family.



    Rack Setup Example:
    2.Rack Diagram.jpg
    Fuel Node:
    3.Fuel Node.JPG.jpg
    Compute and Controller Nodes:
    4.Compute and Controller.JPG.jpg


    InfiniBand configuration


         Note:  All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf). All PKEYs are open on all servers ports (they get it from the SM configurations).
                   The user can choose which VLAN he wants for which network, the mapping between VLAN to PKEY is being done in background.

                   No extra configurations are needed on the switch InfiniBand ports (PKEYs configured on an extra SM server is a pre-requisite).


    InfiniBand SM configuration - Without UFM


    Add/Change the following in the partitions.conf file (Example)

    management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;

    For every network you want to configure in Neutron you have to configure the pkey associated with the VLAN of this network (defined in Neutron).

    vlan1=0x1, ipoib, sl=0, defmember=full : ALL;


    For example:


    We'll have the following configuration of the the partitions.conf file:

    management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;

    vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

    vlan2=0x2, ipoib, sl=0, defmember=full : ALL;

    vlan3=0x3, ipoib, sl=0, defmember=full : ALL;

    vlan4=0x4, ipoib, sl=0, defmember=full : ALL;

    vlan5=0x5, ipoib, sl=0, defmember=full : ALL;

    vlan6=0x6, ipoib, sl=0, defmember=full : ALL;

    vlan7=0x7, ipoib, sl=0, defmember=full : ALL;

    vlan8=0x8, ipoib, sl=0, defmember=full : ALL;

    vlan9=0x9, ipoib, sl=0, defmember=full : ALL;

    vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

    Change the following in /etc/opensm/opensm.conf:

    allow_both_pkeys TRUE

    Restart the openSM

    # /etc/init.d/opensmd restart


    InfiniBand SM configuration - With UFM


    1. Make sure UFM is installed and connected to your fabric.

    2. Edit /opt/ufm/conf/opensm/opensm.conf and change the following values:

        a) set allow_both_pkeys to TRUE (by default allow_both_pkeys are FALSE):

    #Allow both full and limited membership on the same partition

    allow_both_pkeys TRUE

        b) set sm_assign_guid_func to uniq_count (by default sm_assign_guid_func is base_port)

    #SM assigned Alias GUIDs algorithm

    sm_assign_guid_func uniq_count

    3. Edit UFM user extension partitions.conf for overriding default partitioning configuration.

        a) Edit file: /opt/ufm/conf/partitions.conf.user_ext (it should be empty after UFM fresh installation)

        b) add the following line to file which enable both full and limited management pkey:

    management=0xffff,ipoib, sl=0, defmember=both : ALL, ALL_SWITCHES=full,SELF=full;

        c) Add the additional pkeys definitions which are relevant to the specific setup - for example:

    vlan1=0x1, ipoib, sl=0, defmember=full : ALL_CAS;

    vlan2=0x2, ipoib, sl=0, defmember=full : ALL_CAS;

    vlan3=0x3, ipoib, sl=0, defmember=full : ALL_CAS;

    vlan4=0x4, ipoib, sl=0, defmember=full : ALL_CAS;

    vlan5=0x5, ipoib, sl=0, defmember=full : ALL_CAS;

    vlan6=0x6, ipoib, sl=0, defmember=full : ALL_CAS;

    vlan7=0x7, ipoib, sl=0, defmember=full : ALL_CAS;

    vlan8=0x8, ipoib, sl=0, defmember=full : ALL_CAS;

    vlan9=0x9, ipoib, sl=0, defmember=full : ALL_CAS;

    vlan10=0xa, ipoib, sl=0, defmember=full : ALL_CAS;

    vlan10=0xb, ipoib, sl=0, defmember=full : ALL_CAS;

    vlan10=0xc, ipoib, sl=0, defmember=full : ALL_CAS;

    vlan10=0xd, ipoib, sl=0, defmember=full : ALL_CAS;

    vlan10=0xe, ipoib, sl=0, defmember=full : ALL_CAS;

    vlan10=0xf, ipoib, sl=0, defmember=full : ALL_CAS;

    vlan10=0x10, ipoib, sl=0, defmember=full : ALL_CAS;

    vlan10=0x11, ipoib, sl=0, defmember=full : ALL_CAS;

    vlan10=0x12, ipoib, sl=0, defmember=full : ALL_CAS;

    4. Restart UFM

        a. in case of stand-alone UFM:

    # /etc/init.d/ufmd restart

        b. In case of High-availability UFM:

    # /etc/init.d/ufmha restart


    Networks Allocation (Example)

    The example in this post is based on the network allocation defined in this table:
    Admin (PXE)

    The network is used to provision and manage Cloud nodes via the Fuel Master.

    The network is enclosed within a 1Gb switch and has no routing outside.

    This is the default Fuel network.

    This is the Cloud Management network.

    The network uses VLAN 2 over FDR InfiniBand  via SX6036 switch.

    This is the default Fuel network.


    This network is used to provide storage services.

    The network uses VLAN 3 over FDR InfiniBand  via SX6036 switch.

    This is the default Fuel network.

    Public and

    Neutron L3

    Public network is used to connect Cloud nodes to an external network.

    Neutron L3 is used to provide Floating IP for tenant VMs.

    Both networks are represented by IP ranges within same subnet with routing to external networks.

    All Cloud nodes which have Public IP and HA functionality require an additional Virtual IP.

    For our example with 7 Cloud nodes we need 8 IPs in Public network range.

    Consider a larger range if you are planning to add more servers to the cloud later.

    In our build we will use range IP range >> for both Public and Neutron L3. IP allocation will be as follows:

    • Fuel Master IP:
    • Public Range: >> (used for physical servers)
    • Neutron L3 Range: >> (used for Floating IP pool)


    Install the Fuel Master via ISO Image:

    1. Boot Fuel Master Server from the ISO as a virtual CD (click here for the image).
    2. Press the <TAB> key on the very first installation screen which says "Welcome to Fuel Installer" and update the kernel option from showmenu=no to showmenu=yes and hit Enter. It will now install Fuel and reboot the server.
    3. After the reboot, boot from the local disk. The Fuel menu window will start.
    4. Network setup:
      1. Configure eth0 - PXE (Admin) network interface.
        Ensure the default Gateway entry is empty for the interface – the network is enclosed within the switch and has no routing outside.
        Select Apply.
      2. Configure eth1 – Public network interface.
        The interface is routable to LAN/internet and will be used to access the server.
        Configure static IP address, netmask and default gateway on the public network interface.
        Select Apply.
    5. PXE Setup
      The PXE network is enclosed within the switch.
      Do not make changes – proceed with defaults.
      Press Check button to ensure no errors are found.
    6. Time Sync
      Check NTP availability (e.g. via Time Sync tab on the left.
      Configure NTP server entries suitable for your infrastructure.
      Press Check to verify settings.9.Net.NTP.jpg
    7. Navigate to Quit Setup and select Save and Quit to proceed with the installation.
    8. Once the Fuel installation is done, you are provided with Fuel access details both for SSH and HTTP.
      Access Fuel Web UI by Use "admin" for both login and password.

    OpenStack Environment


    Log into Fuel
    1. Open in WEB browser (for example:
    2. Log into Fuel using "admin" for both login and password.
    Creating a new OpenStack environment:
    1. Open a new environment in the Fuel dashboard. A configuration wizard will start.
    2. Configure the new environment wizard as follows:
      • Name and Release
        • Name: TEST
        • Release: Icehouse on CentOS 6.5 (2014.1.1-5.1)
      • Deployment Mode
        • Multi-node with HA
      • Compute
        • KVM
      • Network
        • Neutron with VLAN segmentation
      • Storage Backend
        • Cinder: Default
        • Glance : Default
      • Additional Services
        • None
      • Finish
        • Click Create button
    3. When done, a new TEST environment will be created. Click on it and proceed with environment configuration.

    Configuring the OpenStack Environment:

    Settings Tab

    Kernel parameters
    If you wish to enable iSER or SR-IOV, add this to the list of kernel parameters: intel_iommu=on.
    Mellanox Neutron component
    To work with SR-IOV mode, select Install Mellanox drivers and Mellanox SR-IOV plugin
    Note: The default number of supported virtual functions (VF) is 16. If you want to have more vNICs available, please contact Mellanox Support.
    Configure Storage
    1. To use high performance block storage, check ISER protocol for volumes (Cinder) in the Storage section.
      Note: "Cinder LVM over iSCSI for volumes" should remain checked (default).
    2. Save the settings.
    Public Network Assignment
    1. Make sure Assign public network to all nodes is checked

    Nodes Tab

    Servers Discovery by Fuel
    This section will assign Cloud roles to servers.
    First of all, servers should be discovered by Fuel. For this to happen, make sure the servers are configured for PXE boot over Admin (PXE) network.
    When done, reboot the servers and wait for them to be discovered.
    Discovered nodes will be counted in top right corner of the Fuel dashboard.
    Now you may add UNALLOCATED NODES to the setup.
    First you may add Controller, Storage, and then Compute nodes.
    Verify Infiniband links for nodes
    1. Access the Nodes tab in your environment.
    2. Click on the cog wheel on the right of a node:


       3. Verify that the driver is eth_ipoib and the bus info is an IB interface.



    Add Controller Nodes
    1. Access the Nodes tab in your environment.
    2. Click Add Node.
    3. Identify 3 controller node. Use the last 4 Hexa of its MAC address of interface connected to Admin (PXE) network.
      Assign the node's role to be a Controller node.
    4. Click Apply Changes.
    Add Storage Node
    1. Access the Nodes tab in your environment.
    2. Click Add Node.
    3. Identify your controller node. Use the last 4 Hexa of its MAC address of interface connected to Admin (PXE) network.
      In our example this is an only Supermicro server, so identification is easy.
      Select this node to be a Storage - Cinder LVM node.
    4. Click Apply Changes.
    Add Compute Nodes
    1. Access the Nodes tab in your environment.
    1. Click Add Node.
    2. Select all the nodes that are left and assign them the Compute role.
    3. Click Apply Changes.
    Configure Interfaces
    In this step, we will map each network to a physical interface for each node.
    You can choose and configure multiple nodes in parallel.
    Fuel will not let you to proceed with bulk configuration if HW differences between selected nodes (like the number of network ports) are detected.
    In this case the Configure Interfaces button will have an error icon (see below).
    The example below allows configuring 6 nodes in parallel. The 7th node (Supermicro storage node) will be configured separately.
    1. In this example, we set the Admin (PXE) network to eth0 and the Public network to eth1.
    2. The Storage, Private and Management networks should run on the ConnectX-3 adapters 40/56GbE port.
      This is an example:
    3. Click Back To Node List and perform network configuration for Storage Node
    Note: Port bonding is not supported when using SR-IOV over ConnectX-3 Pro adapter family.
    Configure Disks
    There is no need to change the defaults for Controller and Compute nodes unless you are sure changes are required.
    For the Storage node it is recommended to allocate only high performing RAID as Cinder storage. The small disk shall be allocated to Base System.
    1. Select Storage node
    2. Press Configure Disks button
    3. Click on sda disk bar, set Cinder allowed space to 0 MB and make Base System occupy the entire drive – press USE ALL ALLOWED SPACE.
    4. Press Apply.

    Networks Tab

    Note: In our example, Public network does not use VLAN. If you use VLAN for Public network You should check Use VLAN tagging and set proper VLAN ID.

    In this example, we select VLAN 2 for the management network. The CIDR left untouched.
    In this example, we select VLAN 3 for the storage network. The CIDR is left untouched.
    Neutron L2 Configuration
    In this example, we set the VLAN range to 4-100. It should be aligned with the switch VLAN configuration (above).
    The base MAC is left untouched.
    Neutron L3 Configuration:
    Floating IP range: Configure it to be part of your Public network range, in this example, we select
    Internal Network: Leave CIDR and Gateway with no changes.
    Name servers: Leave DNS servers with no changes.

    Save Configuration
    Click Save Settings at the bottom of page
    Verify Networks
    Click Verify Networks.
    You should see the following message: Verification succeeded. Your network is configured correctly. Otherwise, check the log file for troubleshooting.


    Click the Deploy Changes button and view the installation progress at the nodes tab and view logs.

    Health Test

    1. Click the Health Test tab.
    2. Check the Select All checkbox.
    3. Uncheck Platform services functional tests (image with special packages is required).
    4. Click Run Tests.
    All tests should pass. Otherwise, check the log file for troubleshooting.

    You can now safely use the cloud

    Click the dashboard link at the top of the page.

    Usernames and Passwords:

    • Fuel server Dashboard user / password: admin / admin
    • Fuel server SSH user / password: root / r00tme
    • TestVM SSH user / password: cirros / cubswin:)
    • To get controller node CLI permissions run:  # source /root/openrc



    Prepare Linux VM Image for CloudX:

    In order to have network and RoCE support on the VM, MLNX_OFED (2.2-1 or later) should be installed on the VM environment.

    MLNX_OFED may be downloaded from

    (In case of CentOS/RHEL OS, you can use virt-manager to open existing VM image and perform MLNX_OFED installation).



    Known Issues:

    Issue #
    Configuring more than 16 VFs is not supported by default.
    To have more vNICs available, please contact Mellanox Support.
    This ISO supports only IB SR-IOV installations (regular and HA) over CentOS 6.5 with iSER or iSCSI based storage.
    3Verify network before installation doesn’t support untagged VLAN verification and more than 60 PKEYs (shouldn’t affect installation).
    4Switch SM is not supported.External server based SM node with configurations is required prior to the installation
    5Launching more than 5 VMs at once may cause eswitch allocation problems (eswitchd issue).Delete all VMs and restart eswitchd.
    6In case of partitions.conf changes in the SM machines for "verify network" after failed verification,  bootstrap nodes are not updated with the latest pkeys for another network verification.discovered nodes should be restarted for another network verification after PKEYs change (shouldn’t affect installation).
    7Migration of SR-IOV based VM is not working with Nova Migrate command. Using snapshots.