HowTo Install Mirantis Fuel 6.1 OpenStack with Mellanox Adapters Support (InfiniBand Network)

Version 9

    This post shows how to set up and configure Mirantis Fuel ver. 6.1 (OpenStack Juno based on CentOS 6.5) over InfiniBand 56Gb/s network with support for SR-IOV mode for the VMs on the compute nodes, and in iSER (iSCSI over RDMA) transport mode for the storage nodes.


    For Ethernet 56Gb/s How-to manual, please, refer to the HowTo Install Mirantis Fuel 6.1 OpenStack with Mellanox Adapters Support (Ethernet Network).


    Related references


    Before reading this post, make sure you are familiar with Mirantis Fuel 6.1 installation procedures. It is also recommended to watch HowTo Install Mirantis Fuel 5.1 OpenStack with Mellanox Adapters video.


    Setup Diagram




    Note: All nodes, including Fuel Master Node, should be connected to all five networks.
    Note: Server’s IPMI and the switches management interfaces wiring and configuration are out of scope.


    You need to ensure that there is management access (SSH) to the Mellanox InfiniBand switch to perform the configuration.


    Setup BOM

    Fuel Master, Cloud Controllers and Compute servers:
    • 1 x Fuel Master
    • 3 x Controllers
    • 3 x Computes
    7DELL PowerEdge R620
    • CPU: 2 x E5-2650 @ 2.00GHz
    • MEM: 128 GB
    • HD: 2 x 900GB SAS 10k in RAID-1
    • NIC: Mellanox ConnectX-3Pro VPI (MCX353-FCCT)
    Cloud Storage server1Supermicro X9DR3-F
    • CPU: 2 x E5-2650 @ 2.00GHz
    • MEM: 128 GB
    • HD: 24 x 6Gb/s SATA Intel SSD DC S3500 Series 480GB (SSDSC2BB480G4)
    • RAID Ctrl: LSI Logic MegaRAID SAS 2208 with battery
    • NIC: Mellanox ConnectX-3Pro VPI (MCX353-FCCT)
    Admin  (PXE) and Public switch11Gb switch with VLANs configured to support both networks
    InfiniBand Switch1Mellanox SX1710 SDN 36 port switch configured in InfiniBand mode.

    16 x 1Gb CAT-6e for Admin (PXE) and Public networks

    8 x 56GbE copper cables up to 2m (MC2207130-XXX)


    Note: You can use Mellanox ConnectX-3 Pro EN (MCX313A-BCCT) or Mellanox ConnectX-3 Pro VPI (MCX353-FCCT) adapter cards.

    Note: Please make sure that the Mellanox switch is set as InfiniBand.


    Storage server RAID Setup

    • 2 SSD drives in bays 0-1 configured in RAID-1 (Mirror): The OS will be installed on it.
    • 22 SSD drives in bays 3-24 configured in RAID-10: The Cinder volume will be configured on the RAID drive.

    Network Physical Setup

    1. Connect all nodes to the Admin (PXE) 1GbE switch (preferably through the eth0 interface on board).
      It is recommended to write the MAC address of the Controller and Storage servers to make Cloud installation easier (see Controller Node section below in Nodes tab).

    Note: All cloud servers should be configured to run PXE boot over the Admin (PXE) network.

         2.  Connect all nodes to the Public 1GbE switch (preferably through the eth1 interface on board).

         3.  Connect port #1 (ib0) of ConnectX-3 Pro to InfiniBand switch (Private, Management, Storage networks).


    Note: The interface names (eth0, eth1, p2p1, etc.) may vary between servers from different vendors.

    Note: Port bonding is not supported when using SR-IOV over the ConnectX-3 adapter family.

    Note: Subnet manager (OpenSM) can run on any server that is connected to SwitchX. In this article OpenSM will run on Fuel node.


    Rack Setup Example:

    Rack Setup Example.png


    Fuel, Compute and Controller Nodes:



    Storage Node:


    Networks Allocation (Example)

    The example in this post is based on the network allocation defined in this table:

    Admin (PXE) network is used to provision and manage Cloud nodes by the Fuel Master. The network is enclosed within a 1Gb switch and has no routing outside. is the default Fuel subnet and we use it with no changes.
    Management192.168.0.0/24N/AThis is the Cloud Management network. The network uses VLAN 2 in SX1710 over 40/56Gb interconnect. is the default Fuel subnet and we use it with no changes.
    Storage192.168.1.0/24N/AThis network is used to provide storage services. The network uses VLAN 3 in SX1710 over 40/56Gb interconnect. is the default Fuel subnet and we use it with no changes.
    Public and Neutron L310.7.208.0/2410.7.208.1

    Public network is used to connect Cloud nodes to an external network.

    Neutron L3 is used to provide Floating IP for tenant VMs.

    Both networks are represented by IP ranges within same subnet with routing to external networks.


    All Cloud nodes will have Public IP address. In additional you shall allocate 2 more Public IP addressees ·

    • One IP required for HA functionality ·
    • From Fuel 6.1 virtual router requires additional Public IP address.

    We do not use virtual router in our deployment but still need to reserve Public IP address for it.   So Public Network range is an amount of cloud nodes + 2. For our example with 7 Cloud nodes we need 9 IPs in Public network range.

    Note: Consider a larger range if you are planning to add more servers to the cloud later.

    In our build we will use >> IP range for both Public and Neutron L3.

    IP allocation will be as follows:

    • Fuel Master IP:
    • Public Range: >> (7 used for physical servers, 1 reserved for HA and 1 reserved for virtual router)
    • Neutron L3 Range: >> (used for Floating IP pool)


    Below scheme illustrates our setup IP allocation.



    Install the Fuel Master from ISO Image:

    Note: Try to avoid start of other nodes except fuel master until Mellanox Plugin will be installed.


    1. Boot Fuel Master Server from the ISO as a virtual DVD (click here for the image).

    2. After the reboot, boot from the local disk. The Fuel menu window will start.

    3. Network setup:

    1. Configure eth0 - PXE (Admin) network interface.
      Ensure the default Gateway entry is empty for the interface – the network is enclosed within the switch and has no routing outside. Select Apply.
    2. Configure eth1 – Public network interface.
      The interface is routable to LAN/internet and will be used to access the server. Configure static IP address, netmask and default gateway on the public network interface. Select Apply.

    4. Set the PXE Setup. 
         The PXE network is enclosed within the switch. Do not make changes – proceed with defaults. Press Check button to ensure no errors are found.

    5. Set the Time Sync.

    • Check NTP availability (e.g. via Time Sync tab on the left.
    • Configure NTP server entries suitable for your infrastructure.
    • Press Check to verify settings.


    6. Proceed with the installation.
         Navigate to Quit Setup and select Save and Quit.


        Once the Fuel installation is done, you are provided with Fuel access details both for SSH and HTTP.
    7. Access Fuel Web UI by Use "admin" for both login and password.
        For SSH access use username root and password r00tme

    Install Mellanox Plugin

    Mellanox plugin configures support for Mellanox ConnectX-3 Pro network adapters, enabling high-performance SR-IOV compute traffic networking, iSER (iSCSI) block storage networking which reduces CPU overhead, boosts throughput, reduces latency, and enables network traffic to bypass the software switch layer.


    Follow below steps to install the plugin. For the complete instructions, please refer to: HowTo Install Mellanox OpenStack Plugin for Mirantis Fuel 6.1)

    1. Download the Mellanox plugin rpm from here and store it on your Fuel Master server.

    2. Log into the fuel master and run below command from the directory with the plugin:


    Note: The Mellanox plugin replaces the current bootstrap image, the original image is backed up in /opt/old_bootstrap_image/

    3. Verify that the plugin was successfully installed.
        It should be displayed when running the fuel plugins command.

    4. Reboot of all discovered nodes.
         Run the # fuel nodes command to check if there are any discovered nodes already.
        Use buildin command # reboot_bootstrap_nodes to reboot discovered nodes.


    InfiniBand Configuration

    Since OpenSM runs on Fuel master, OFED should be installed on this server and configure PKEYs in partition.conf file. No extra configurations are required on the switch side.

    Note: All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf). All PKeys are open on all servers ports (they get it from the SM configurations). The user can choose which VLAN he wants for which network, the mapping between VLAN to PKEY is done in the background.

    1. Download MLNX_OFED 3.1 TGZ archive from MLNX_OFED Download Center to the Fuel node into /tmp directory


    2. Install MLNX_OFED 3.1.


        a. Install dependencies for MLNX_OFED.

    # yum install kernel-devel python-devel redhat-rpm-config rpm-build gcc gtk2 atk cairo gcc-gfortran tcsh tcl bc tk


        b. Go to /tmp folder and unpack the downloaded TGZ archive.

    # tar -xzvf MLNX_OFED_LINUX-3.1-1.0.3-rhel6.5-x86_64.tgz

       c. Go to the folder where MLNX_OFED was unpacked.

    # cd MLNX_OFED_LINUX-3.1-1.0.3-rhel6.5-x86_64

       d. Build customized MLNX_OFED binaries to support Fuel server kernel.

    # ./ -m ./ --make-tgz

      e. Unpack TGZ archive that you just created.

    # tar -xzvf MLNX_OFED_LINUX-3.1-1.0.3-rhel6.5-x86_64-ext.tgz

       f. Go to the folder where MLNX_OFED was unpacked.

    # cd MLNX_OFED_LINUX-3.1-1.0.3-rhel6.5-x86_64-ext

      g. Install MLNX_OFED.

    # mlnxofedinstall –all

      h. Reboot Fuel Master or use command:

    # /etc/init.d/openibd restart

    For more information, please refer to the official how-to.


    InfiniBand OpenSM Configuration


    We are going to create two files on Fuel node: /etc/opensm/partitions.conf and /etc/opensm/opensm.conf.

    1. Create a new opensm.conf file.

    # opensm -c /etc/opensm/opensm.conf

    2. Enable virtualization by editing /etc/opensm/opensm.conf and changing the allow_both_pkeys value to TRUE.

    allow_both_pkeys TRUE

    3. Define the partition keys which are analog for Ethernet VLAN. Each VLAN will be mapped to one PK.
         Add/Change the following with the command #vi /etc/opensm/partitions.conf file (Example)

    management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;

    vlan1=0x1, ipoib, sl=0, defmember=full : ALL;

    vlan2=0x2, ipoib, sl=0, defmember=full : ALL;

    vlan3=0x3, ipoib, sl=0, defmember=full : ALL;

    vlan4=0x4, ipoib, sl=0, defmember=full : ALL;

    vlan5=0x5, ipoib, sl=0, defmember=full : ALL;

    vlan6=0x6, ipoib, sl=0, defmember=full : ALL;

    vlan7=0x7, ipoib, sl=0, defmember=full : ALL;

    vlan8=0x8, ipoib, sl=0, defmember=full : ALL;

    vlan9=0x9, ipoib, sl=0, defmember=full : ALL;

    vlan10=0xa, ipoib, sl=0, defmember=full : ALL;

    . . .

    vlan100=0x64, ipoib, sl=0, defmember=full : ALL;

    In this example:

    • VLAN2 assigned to PK 0x2 and will be used for Openstack Management network
      VLAN3 assigned to PK 0x3 and will be used for Openstack Storage network
      VLANs 4 through 100 are assigned to PKs 0x4 to 0x64 will be used for Tenant networks


    Note: VLAN1 is defined, but not used for consistency with Ethernet setup installation.

    Note: Maximum amount of VLANs is limited to 127.


    4. Restart the OpenSM

    # /etc/init.d/opensmd restart



    Creating OpenStack Environment

    Log into Fuel

    1. Open a browser (for example:

    2. Log into Fuel using admin/admin for login/password.


    Creating a new OpenStack Environment:


    1. Open a new environment in the Fuel dashboard. A configuration wizard will start.

    2. Configure the new environment wizard as follows:

    • Name and Release
      1. Name: TEST
      2. Release: Icehouse on CentOS 6.5 (2014.1.1-5.1)
    • Compute
      1. KVM
    • Network
      1. Neutron with VLAN segmentation
    • Storage Backend
      1. Cinder: Default
      2. Glance : Default
    • Additional Services
      1. None
    • Finish
      1. Click Create button
      2. When done, a new TEST environment will be created. Click on it and proceed with environment configuration.

    Configuring the OpenStack Environment

    Settings Tab

    Mellanox Neutron and iSER storage component Public Network Assignment


    To work with SR-IOV mode select Mellanox Openstack features, Neutron SR-IOV plugin.

    To work with iSER transport – select Mellanox Openstack features (if not selected before), iSER protocol for volumes (Cinder)


    Note: By default, the number of virtual NICs is 16. Please contact Mellanox support if you want to use more.




    Public Network Assignment

    Make sure Assign public network to all nodes is checked


    Save settings.



    Nodes Tab


    Servers Discovery by Fuel

    This section will show how to assign Cloud roles to servers. To do so, the servers should be discovered by Fuel. Therefore, the servers must be configured for PXE boot over Admin (PXE) network. When done, reboot the servers and wait for them to be discovered. Discovered nodes will be counted in top right corner of the Fuel dashboard.


    Now you may add UNALLOCATED NODES to the setup.
    First you may add Controller, Storage, and then Compute nodes.


    Verify Infiniband links for nodes

    1. Access the Nodes tab in your environment.

    2. Click on the cog wheel on the right of a node:


    3. Verify that the driver is eth_ipoib and the bus info is an IB interface.


    Add Controller Nodes

    1. Click Add Node.

    2. Identify 3 controller node. Use the last 4 Hexadecimal digits of its MAC address of interface connected to Admin (PXE) network. Assign the node's role to be a Controller node.

    3. Click Apply Changes button


    Add Storage Node

    1. Click Add Node.

    2. Identify your storage node. Use the last 4 Hexadecimal digits of its MAC address of interface connected to Admin (PXE) network. In our example this is the only Supermicro server, so identification by vendor is easy. Select this node to be a Storage - Cinder node.

    3. Click Apply Changes button

    Add Compute Nodes

    1. Click Add Node.

    2. Select all the nodes that are left and assign them the Compute role.

    3. Click Apply Changes.


    Configure Interfaces

    In this step, we will map each network to a physical interface for each node.You can choose and configure multiple nodes in parallel.

    Fuel will not let you to proceed with bulk configuration if HW differences between selected nodes (like the number of network ports) are detected.

    In this case the Configure Interfaces button will have an error icon (see below).



    The example below allows configuring 6 nodes in parallel. The 7th node (Supermicro storage node) will be configured separately.


    1. In this example, we set the Admin (PXE) network to eth0 and the Public network to eth1.

    2. The Storage, Private and Management networks should run on the ConnectX-3 adapters 40/56GbE port. This is an example:

    3. Click Back To Node List and perform network configuration for Storage Node

    Note: Port bonding is not supported when using SR-IOV over ConnectX-3 Pro adapter family.
    Configure Disks There is no need to change the defaults for the Controller and Compute nodes unless you are sure changes are required.
    For the Storage node it is recommended to allocate only high performing RAID as Cinder storage.
    The small disk shall be allocated to Base System.

    4. Select Storage node


    5. Press Configure Disks button


    6. Click on sda disk bar, set Cinder allowed space to 0 MB and make Base System occupy the entire drive – press USE ALL ALLOWED SPACE.


        1. Press Apply.

    Networks Tab



    Note: In our example, Public network does not use VLAN. If you use VLAN for Public network, you should check Use VLAN tagging and set proper VLAN ID.

    Public IP range: Consists of nodes IP range and Floating IP range. In our case, the full range is

    IP range: Configure it to be part of your Public IP range, in this example, we select

    Floating IP range: Configure it to be part of your Public IP range, in this example, we select




    In this example, we select VLAN 3 for the storage network. The CIDR is left untouched.





    In this example, we select VLAN 2 for the management network. The CIDR left untouched.



    Neutron L2 Configuration

    In this example, we set the VLAN range to 4-100. It should be aligned with the switch VLAN configuration (above).
    The base MAC is left untouched.



    Neutron L3 Configuration:

    Internal Network: Leave CIDR and Gateway with no changes.

    Name servers: Leave DNS servers with no changes.


    Save Configuration

    Click Save Settings at the bottom of page


    Verify Networks

    Click Verify Networks.
    You should see the following message: Verification succeeded. Your network is configured correctly. Otherwise, check the log file for troubleshooting.


    Note: If your public network has running DHCP server, you can face verification failure as below.

    If the range, selected for the cloud above is not overlapping with DHCP pool, you can ignore this message.

    If overlap exists, please fix it



    Click the Deploy Changes button and view the installation progress at the nodes tab and view logs.


    Stage 1: OS installation



    Stage 2: OS installation is finished



    Stage 3: Install OPSTK on the 1st controller



    Stage 4: Install OPSTK on the rest the controllers



    Stage 5: Install OPSTK on the Compute and Storage nodes



    Installation is completed


    Health Test

    1. Click the Health Test tab.

    2. Check the Select All checkbox.

    3. Un-check Platform services functional tests (image with special packages is required).

    4. Un-check Launch instance, create snapshot, launch instance from snapshot in Functional tests group.  (This is known issue that snapshot creation of running SRIOV instance fails.  See Known Issues for more details)

    5. Click Run Tests.


    All tests should pass. Otherwise, check the log file for troubleshooting.


    You can now safely use the cloud

    Click the dashboard link at the top of the page.



    Usernames and Passwords:

    • Fuel server Dashboard user / password: admin / admin
    • Fuel server SSH user / password: root / r00tme
    • TestVM SSH user / password: cirros / cubswin:)
    • To get controller node CLI permissions run:  # source /root/openrc

    Prepare Linux VM Image for CloudX:

    In order to have network and RoCE support on the VM, MLNX_OFED (2.2-1 or later) should be installed on the VM environment.

    MLNX_OFED may be downloaded from

    (In case of CentOS/RHEL OS, you can use virt-manager to open existing VM image and perform MLNX_OFED installation).


    Known Issues:

    Issue #DescriptionWorkaround
    1Configuring more than 16 VFs is not supported by default.To have more vNICs available, please contact Mellanox Support.
    2Verify network can cause error message if public network has running DHCP serverIf the range, selected for the cloud above is not overlapping with DHCP pool, please ignore this message. If overlap exists, please fix it.
    3Verify network before installation doesn’t support untagged VLAN verification and more than 60 PKEYs (shouldn’t affect installation).
    4Switch SM is not supported.External server based SM node with configurations is required prior to the installation
    5Launching more than 5 VMs at once may cause eswitch allocation problems (eswitchd issue).Delete all VMs and restart eswitchd.
    6In case of partitions.conf changes in the SM machines for "verify network" after failed verification, bootstrap nodes are not updated with the latest pkeys for another network verification.Discovered nodes should be restarted for another network verification after PKEYs change (shouldn’t affect installation).
    7Migration of SR-IOV based VM is not working with Nova Migrate command.Using snapshots.
    8Amount of VLANs is limited to 127