HowTo Install Mirantis OpenStack 8.0 with Mellanox ConnectX-4 Adapters Support (InfiniBand Network at Scale)

Version 7

    This post describes how to setup and configure Mirantis Openstack 8.0 (Liberty based on Ubuntu 14.04 LTS) over an InfiniBand 100Gb/s network with

    PCI-SIG Single Root I/O Virtualization (SR-IOV) Virtual Machine (VM) network mode and iSER (iSCSI over RDMA) transport mode for block storage.


    To access the Ethernet 100Gb/s How-to manual, refer to the CloudX with MOS 8.0 over Mellanox ConnectX-4 Adapters (Ethernet Network + BOND + SR-IOV + VLAN Segmentation).

    Related References


    Before reading this post, make sure you are familiar with Mirantis Openstack 8.0 installation procedures.


    Before starting to use the Mellanox switches, we recommend that you upgrade the switches to the latest MLNX-OS version.



    In this example we are setting up an example for scaling in the in future.

    Main highlights of this example are:

    • The Non-blocking Fat-Tree Network schema with two cores and two edge switches that support 36* nodes.
    • The configuration can scale up to 18 core and 36 edge switches and support 648 nodes, as shown in the diagram below.
    • Three controller nodes use HA mode.
    • All nodes are connected to Admin (PXE), Public, Private, Management and Storage networks.
    • Unified Network Manager (UFM) is running on Deployment node. UFM runs the subnet manager to manage all network devices in the fabric.

    • The Fuel master is running as the VM in Deployment node.
    • NEO is running as VM in Deployment node and is used as the communication gateway between the UFM and ML2 plugin.


    * Deployment node is also connected to the fabric. Note that the number of Cloud nodes is limited to 35.


    You can use more controllers but the amount should always be an odd number.

    If you plan to use three or more racks, you should consider distributing Controller nodes in different racks to increase redundancy.

    Note: Instructions that describes the server’s IPMI wiring and configuration are out of the scope of this post.


    Setup Diagram

    The following illustration shows the example configuration.



    Setup Hardware Requirements

    Deployment node1

    Select a non-high performing server to run Fuel VM and UFM software with the following:


    CPU: Intel E5-26xx or later model

    HD: 250 GB or larger

    RAM: 32 GB or more


    • 2 x 1Gb
    • Mellanox ConnectX-4 VPI dual port (MCX456A-ECAT) or single port (MCX455A-ECAT) NIC
    Intel Virtualization technologies must be enabled in BIOS.
    If you store repositories on Fuel VM locally, you need a 350GB HD or larger.
    Cloud Controllers and compute servers:
    • 3 x Controllers
    • 3 x Computes

    Strong servers to run Cloud control and Tenant’s VM workload that meet the following:


    CPU: Intel E5-26xx or later model

    HD: 450 GB or larger

    RAM: 128 GB or more


    • 2 x 1Gb
    • Mellanox ConnectX-4 VPI dual port (MCX456A-ECAT) or single port (MCX455A-ECAT) NIC
    Note: Intel Virtualization technologies and SRIOV must be enabled in BIOS
    Cloud Storage server1

    A high IO performance strong server to act as Cinder Volume backend featuring:


    CPU: Intel E5-26xx or later model


    • OS: 100 GB or larger
    • Cinder Volume: SSD drives configured in RAID-10 for best performance

    RAM: 64 GB or more


    • 2 x 1Gb
    • Mellanox ConnectX-4 VPI dual port (MCX456A-ECAT) or single port (MCX455A-ECAT) NIC
    Note: Intel Virtualization technologies and SRIOV must be enabled in BIOS
    Admin (PXE) and Public switch11Gb switch with VLANs configured to support both networks.
    InfiniBand Switch4Mellanox SB7700 36 ports 100Gb/s switch.
    • 16 x 1Gb CAT-6e for Admin (PXE) and Public networks
    • 16 x 100Gb VPI copper cables up to 2m (MCP1600-EXXX)


    Note: We configure eight nodes only, so there is no need to wire more than eight edge-to-core uplinks to keep fabric non-blocking.

    If you add more nodes you need to increase edge-to-core uplinks accordingly.

    Server Configuration

    In order for Cloud hardware to work, you should meet several prerequisites.

    Go to the BIOS of each node and ensure that :

    • Either Intel VT or AMD-V (depending on your CPU type) virtualization is enabled on all nodes, including Deployment node.
    • SR-IOV is enabled on all Controller, Compute and Storage nodes.
    • All nodes except Deployment node are configured to boot from the NIC connected to Admin (PXE ) network.



    Storage Server RAID Setup

    • Two drives in bays 0-1 configured in RAID-1 (Mirror): The OS and Openstack will be installed on it.
    • Twenty-two SSD drives in bays 3-24 configured in RAID-10: The Cinder volume will be configured on the RAID-10 volume.


    Physical Network Setup

    1. Connect all nodes to the Admin (PXE) 1GbE switch, preferably through the eth0 interface on the board. We recommend that you record the MAC address of the Controller and Storage servers to make Cloud installation easier (see Controller Node section below in the Nodes tab).

    2. Connect all nodes to the Public 1GbE switch, preferably through the eth1 interface on board.

    3. Connect Port #1 (ib0) of ConnectX-4 of all nodes (including Deployment node) to an InfiniBand switch (Private, Management, and Storage networks).

      Note: The interface names (eth0, eth1, p2p1, etc.) can vary between servers from different vendors.

    Rack Setup Example

    Deployment Node

    Compute and Controller Nodes

    Storage Node
    This is the same as Compute and Controller nodes.



    Networks Allocation (Example)

    The example in this post is based on the network allocation defined in this table:

    Admin (PXE)

    The network is used to provision and manage Cloud nodes by the Fuel Master. The network is enclosed within a 1Gb switch and has no routing outside. is the default Fuel subnet and we use it with no changes.

    Management192.168.0.0/24N/AThis is the Cloud Management network. The network uses VLAN 2 in SB7700 over 100Gb interconnect. is the default Fuel subnet and we use it with no changes.
    Storage192.168.1.0/24N/AThis network is used to provide storage services. The network uses VLAN 3 in SB7700 over 100Gb interconnect. is the default Fuel subnet and we use it with no changes.
    Public and Neutron L310.7.208.0/2410.7.208.1Public network is used to connect Cloud nodes to an external network.Neutron L3 is used to provide Floating IP for tenant VMs.Both networks are represented by IP ranges within same subnet with routing to external networks.
    All Cloud nodes will have Public IP address. In additional you shall allocate 2 more Public IP addressees ·
    • One IP required for HA functionality
    • Virtual router requires an additional Public IP address

    We do not use a virtual router in our deployment but still need to reserve Public IP address for it. So Public Network range is an amount of cloud nodes + 2. For our example with seven Cloud nodes we need nine IPs in the Public network range.

    Note: Consider a larger range if you plan to add more servers to the cloud later.

    In our build we use >> IP range for both Public and Neutron L3.

    IP allocation will be as follows:

    • Deployment node:
    • Fuel Master IP:
    • NEO VM IP:
    • Public Range: >> (7 used for physical servers, one reserved for HA and one reserved for virtual router)
    • Neutron L3 Range: >> (used for Floating IP pool)


    The scheme below illustrates our setup IP allocation.


    Install the Deployment Server

    In our setup we install CentOS release 7.2 (a 64-bit distribution). We used the CentOS-7-x86_64-Minimal-1511.iso image and installed the Minimal configuration. We will install all missing packages later.

    Two 1Gb interfaces are connected to Admin (PXE) and Public networks:

      • em1 (first interface) is connected to Admin (PXE) and configured statically.
        The configuration will not actually be used, but will save time on bridge creation later.
        • IP:
        • Netmask:
        • Gateway: N/A
      • em2 (second interface) is connected to Public and configured statically:
        • IP:
        • Netmask:
        • Gateway:


    Note: Install all OS updates and restart the server before you proceed.


    InfiniBand Configuration

    Since Unified Fabric Manager (UFM) runs in the Deployment node, Mellanox OFED should be installed on this server. No extra configurations are required on the switch side.


    Download the Mellanox OFED Driver

    Download MLNX_OFED version 3.2- of the ISO file from MLNX_OFED Download Center to the Deployment node into /tmp directory.

    Warning: Only this version of OFED should be installed, or UFM installation (on later steps) might fail!


    Install MLNX_OFED Version 3.2

    1. Install dependencies for MLNX_OFED.
      # yum install gtk2 tcl gcc-gfortran tk perl createrepo kernel-devel python-devel pciutils lsof redhat-rpm-config rpm-build libnl tcsh
    2. Create a new folder to mount the ISO image  just downloaded.
      # mkdir /tmp/ofed
    3. Go to /tmp folder and mount the downloaded ISO image file.

      # mount -o loop /tmp/MLNX_OFED_LINUX-3.2- /tmp/ofed/
    4. Go to the folder where MLNX_OFED was unpacked.

      # cd /tmp/ofed
    5. Since CentOS 7.2 is not supported by this version of OFED, we need to build custom OFED binaries.To do that, run the following:
      # ./ -m ./ --make-tgz
    6. Unpack the file you just created.

      # tar -zxvf /tmp/MLNX_OFED_LINUX-3.2-
    7. Go to newly-created folder and install MLNX_OFED as follows:

      # cd /tmp/MLNX_OFED_LINUX-3.2-
      # ./mlnxofedinstall --all
    8. Reboot the Deployment node or use the following command:

      # /etc/init.d/openibd restart

    For more information, please refer to the MLNX_OFED manual.


    Downloading UFM

    Note that UFM is not free product, but it is available as a 30-day free trial. For our purposes you will use the trial version, which can be downloaded from Mellanox website by requesting a download link from here:

    Follow the instructions on the page to obtain the UFM with an evaluation license or contact reseller in your region.


    Installing UFM

    1. Disable the Network Manager.
      # sudo systemctl stop NetworkManager.service
      # sudo systemctl disable NetworkManager.service
    2. Disable the SELinix service.
      # vi /etc/sysconfig/selinux

      Change line  SELINUX=enforcing to SELINUX=disabled

    3. Reboot to apply changes.

    4. Install prerequisite packages.
      # yum install httpd mariadb mariadb-server php MySQL-python net-snmp net-snmp-libs net-snmp-utils mod_ssl pexpect telnet pyOpenSSL libxslt unixODBC bc net-tools
    5. Remove conflicting package versions:
      # yum remove opensm-4.6.1.MLNX20160112.774e977-0.1.32101.x86_64 kernel-mft-4.3.0-3.10.0_327.28.2.el7.x86_64.x86_64 mft-4.3.0-25.x86_64
    6. Configure the  ib0 interface.
      UFM requires that the IP is configured on the ib0 interface. You can use any IP that does not overlap with cloud networks (which are and, by default).
      # vi /etc/sysconfig/network-scripts/ifcfg-ib0

    7. Complete the UFM installation.
      Copy the downloaded installation file to /tmp folder and run command:
      # tar xzvf ufm-5.6.0-10.el7.x86_64.tgz
      Since this installation does not support current system kernel, you must compile the missing kernel-dependent MFT RPM.
      a. Create temporary folder:
      # mkdir /tmp/ufm.tmp
      b. Copy MFT sources to this folder:
      # cp /tmp/ufm-5.6.0-10.el7.x86_64/ufm-rpm-src.tar.gz /tmp/ufm.tmp
      c. Go to the temporary folder and unpack MFT source files.

      # cd /tmp/ufm.tmp/

      # tar -zxvf ufm-rpm-src.tar.gz
      d. Compile MFT modules.
      # ./ -o /tmp/ufm-5.6.0-10.el7.x86_64 -m
      e. Return to the unpacked installation and run:
      # cd /tmp/ufm-5.6.0-10.el7.x86_64
      # ./ -o ib


    Configuring UFM

    You need to enable virtualization and allow all Partition Keys (PKs):

      1. Edit the config file and configure the following parameters described below:

        # vi /opt/ufm/conf/opensm/opensm.conf

        virt_enabled 2

        no_partition_enforcement TRUE

        part_enforce off

        allow_both_pkeys FALSE

      2. Define the partition keys, which are analog for Ethernet VLANs. Each VLAN will be mapped to one PKey. Add/Change the following with the command file as the example shows:

        # vi /opt/ufm/conf/partitions.conf.user_ext

        vlan1=0x1, ipoib, sl=0, defmember=full: ALL_CAS;

        # Storage and management vlan should be define as follow

        vlan2=0x2, ipoib, sl=0, defmember=full: ALL_CAS;

        vlan3=0x3, ipoib, sl=0, defmember=full: ALL_CAS;

        # define OpenSM as a member for all OpenStack vlans. If not guest will have link down on “ibdev2netdev” and no connectivity.

        vlan4=0x4, ipoib, sl=0, defmember=full: SELF;

        vlan5=0x5, ipoib, sl=0, defmember=full: SELF;

        vlan6=0x6, ipoib, sl=0, defmember=full: SELF;

        vlan7=0x7, ipoib, sl=0, defmember=full: SELF;

        vlan8=0x8, ipoib, sl=0, defmember=full: SELF;

        vlan9=0x9, ipoib, sl=0, defmember=full: SELF;

        vlan10=0xa, ipoib, sl=0, defmember=full: SELF;

        vlan11=0xb, ipoib, sl=0, defmember=full: SELF;

        vlan12=0xc, ipoib, sl=0, defmember=full: SELF;


        vlan128=0x80, ipoib, sl=0, defmember=full: SELF;

      3. Restart the UFM service.

        # /etc/init.d/ufmd restart


    Configure Deployment Node for Running Fuel and NEO Appliance VMs

    Log in to Deployment node by SSH or locally and perform actions listed below:

      1. Install the packages required for virtualization.
        # sudo yum install qemu-kvm libvirt libvirt-python libguestfs-tools virt-install virt-manager docker-io lvm2
      2. Install packages required for x-server.
        # sudo yum install xorg-x11-xauth xorg-x11-server-utils xclock
      3. Reboot the Deployment server.


    Create and Configure a New VM To Run the NEO Appliance Node

      1. Download the image of NEO VM from Mellanox customer portal (see the link at the top of this page).
      2. Save this image to the Deployment node.
      3. Start the virt-manager.
      4. Create a new VM using the virt-manager wizard.
      5. During the creation process provide VM with four cores and 8GB of RAM.
      6. During the creation process select the saved NEO image as an existing disk.
      7. Configure the network so the NEO VM has a NIC connected to the Switch Management network.
        Use virt-manager to create bridges br-em2 - Public network bridge to connect NEO VM's eth0 to the Public network.
      8. Connect the NEO VM eth0 interface to br-em2. The illustration below shows the relationship of the Deployment node to the Public network.
      9. Save the settings and start the VM.


    Configuring NEO

    The default NEO OS login/password are root/123456.

    1. By default, NEO has configured eth0 interface dynamically. Assign it a static IP address for the Public network.
    2. Integrate NEO with UFM.
      Edit the config
      file and configure the IP address and credentials for UFM.
      # vi /opt/neo/providers/ib/conf/netservice.cfg
      [UFM IB Params]
      #ip = ip,ip,...
      ip =
      #user = user,user,...
      user = admin
      #password = password,password,...
      password = 123456
    3. Restart the NEO service.
      # /opt/neo/neoservice restart


    Create and Configure a New VM To Run the Fuel Master Installation

    1. Start virt-manager.
    2. Create a new VM using the virt-manager wizard.
    3. During the creation process provide VM with four cores, 8GB of RAM, and 200GB disk.
      Note: For details see Fuel Master node hardware requirements section in Mirantis OpenStack Planning Guide — Mirantis OpenStack v8.0 | Documentation.
    4. Mount the Fuel installation disk to VM virtual CD-ROM device.
    5. Configure the network so the Fuel VM has two NICs connected to Admin(PXE) and Public networks.
      1. Use virt-manager to create bridges.
        1. br-em1 - Admin(PXE) network bridge.
          It is used to connect Fuel VM's eth0 to Admin (PXE) network.
        2. br-em2 - Public network bridge
          It is used to connect Fuel VM's eth1 to Public network.
      2. Connect Fuel VM eth0 interface to br-em1.
      3. Add to Fuel VM eth1 network interface and connect it to br-em2 shown below:

        Note: You can define any other names for bridges. In this example names were defined according to the names of connected physical interfaces for the Deployment node.
    6. Save settings and start VM.


    Install the Fuel Master from ISO Image:

    Note: Avoid starting the other nodes except for the Fuel Master until the Mellanox plugin is installed.
    1. Boot Fuel Master Server from the ISO image as a virtual DVD (click here to download ISO image).
    2. Choose option 1. and press TAB button to edit default options:

      a. Remove the default gateway (
      b. Change DNS to (the Fuel VM IP).
      c. Add to the end "showmenu=yes".
      The tuned boot parameters shall look like this:

      Note: Do not try to change eth0 to another interface or deployment may fail.
    3. The Fuel VM will reboot itself after initial installation is completed and the Fuel Menu appears.
      Note: Ensure that VM will start from the Local Disk and not the CD-ROM. Otherwise you will start the installation process from the beginning.
    4. Complete the network setup:
      1. Configure eth0 - PXE (Admin) network interface.
        Ensure the default Gateway entry is empty for the interface – the network is enclosed within the switch and has no routing outside.
        Select Apply.
      2. Configure eth1, the Public network interface.
        The interface is routable to LAN/internet and will be used to access the server. Configure the static IP address, netmask and default gateway on the public network interface.
        Select Apply.
    5. Set the PXE Setup.
      The PXE network is enclosed within the switch. Do not make any changes. Use the default settings.

      Press the Check button to ensure no errors are found.
    6. Set the Time Sync.
      Choose the Time Sync tab on the left menu.
      Configure NTP server entries suitable for your infrastructure.
      Press Check to verify settings.
    7. Proceed with the installation.
      Navigate to Quit Setup and select Save and Quit.

      Once the Fuel installation is done, you are provided with Fuel access details both for SSH and HTTP.
    8. Configure the Fuel Master VM SSH server to allow connections from Public network.
      By default Fuel will accept SSH connections only from the Admin(PXE) network.
      Follow the steps to allow connections from Public Network:
      1. Use virt-manager to access Fuel Master VM console.
      2. Edit sshd_config:
        # vi /etc/ssh/sshd_config
      3. Find and comment line:
      4. Restart sshd:
        # service sshd restart
    9. Access Fuel by:
      • Web UI by (use admin/admin as  user/password)
      • SSH by connect to (use root/r00tme as user/password)


    Maintenance Updates

    It is highly recommended that you install Mirantis Fuel maintenance updates before proceeding with Cloud deployment.

    Please read Mirantis Openstack v8.0 — Maintenance Updates section to know how to apply updates.


    Install Mellanox Plugin

    The Mellanox plugin configures support for Mellanox ConnectX-4 network adapters, enabling high-performance SR-IOV compute traffic networking, iSER (iSCSI) block storage networking which reduces CPU overhead, boosts throughput, reduces latency, and enables network traffic to bypass the software switch layer.


    Follow the steps below to install the plugin. For the complete instructions, please refer to: HowTo Install Mellanox OpenStack Plugin for Mirantis Fuel 8.0).


    Install the Fuel Master node. For more information on how to create a Fuel Master node, please see Mirantis Fuel 8.0 documentation.

    1. Download the plugin rpm file for MOS8.0 from Mellanox page and store it on your Fuel Master server.

    2. To copy the plugin on the already-installed Fuel Master node, use the scp command as follows:

      # scp mellanox-plugin-3.2-3.2.1-1.noarch.rpm root@<Fuel_Master_ip>:/tmp
    3. Install plugin from /tmp directory:
      # cd /tmp
      # fuel plugins --install mellanox-plugin-3.2-3.2.1-1.noarch.rpm
      Note: The Mellanox plugin replaces the current bootstrap image, and the original image is backed up in /opt/old_bootstrap_image/.
    4. Verify that the plugin was successfully installed. It should be displayed when running the fuel plugins command.
      [root@fuel ~]# fuel plugins
      id | name            | version | package_version
      1  | mellanox-plugin | 3.2.1   | 3.0.0
    5. Create new bootstrap image by running create_mellanox_bootstrap --link_type {eth,ib,current} --max_num_vfs={amount_VFs}.
      For IB setup run:
      [root@fuel ~]# create_mellanox_bootstrap --link_type ib --max_num_vfs=16
      Try to build image with data:
      certs: null
      container: {format: tar.gz, meta_file: metadata.yaml}
      . . .
      . . .
      . . .
      Bootstrap image f790e9f8-5bc5-4e61-9935-0640f2eed949 has been activated.
      Note that 16 is the default number of Virtual Functions that can be configured for use. You can use the  --max_num_vfs parameter to change the default value.
      This amount will be burned into Mellanox ConnectX-4 firmware.
      Note: Maximum supported value is 31. Please consult with Mellanox support if you want to use more than 16 VF.
    6. Reboot all, even already-discovered nodes.
      You can perform this step manually or use the command described below to reboot already-discovered nodes.
      # reboot_bootstrap_nodes -a


    Creating a New OpenStack Environment

    Open in WEB browser (for example: and log into Fuel environment using admin/admin as the username and password.


    1. Open a new environment in the Fuel dashboard. A configuration wizard will start.
    2. Configure the new environment wizard as follows:
        • Name and Release
          • Name: SR-IOV IB
          • Release: Liberty on Ubuntu 14.04
        • Compute
          • QEMU-KVM
        • Network
          • Neutron with VLAN segmentation
        • Storage Backend
          • Block storage: LVM
        • Additional Services
          • None
        • Finish
          • Click Create button
    3. Click on the new environment just created and proceed with environment configuration.


    Configuring the OpenStack Environment

    Settings Tab

    1. Enable KVM hypervisor type.
      KVM is required to enable Mellanox Openstack features.

      Open the Settings tab, select Compute section and then choose KVM as the hypervisor type.

    2. In the General section add intel_iommu=on to kernel parameters.

    3. Enable desired Mellanox OpenStack features.
      1. Open the Other section.
      2. Select relevant Mellanox plugin versions if you have multiple versions installed.

      3. Enable SR-IOV support.
        1. Check SR-IOV direct port creation in private VLAN networks (Neutron).
        2. Set the desired number of Virtual NICs.

          Note: This note is relevant for VLAN segmentation only.

          : Number of virtual NICs is amount of virtual functions (VFs) that will be available on Compute node.
          It should not exceed value used during bootstrap creation (default is 16).

          : One VF will be utilized for iSER storage transport if you choose to use iSER. In this case you will get 1 VF less for Virtual Machines.
        3. NEO SDN controller auto VLAN Provisioning (Neutron)
          If selected, NEO SDN agent will be enabled on the nodes and PKeys will be assigned only to the required ports
        4. iSER protocol for volumes (Cinder).
          By enabling this feature you will use iSER block storage transport instead or ISCSI.
          ISCSI Extension over RDMA (iSER) improves latency, improves bandwidth and reduces CPU overhead.
          Note: A dedicated Virtual Function will be reserved for a storage endpoint, and the priority flow control has to be enabled on the switch side port.


    Nodes Tab

    Servers Discovery by Fuel

    This section assigns Cloud roles to servers. Servers should be discovered by Fuel, hence, make sure the servers are configured for PXE boot over Admin (PXE) network. When done, reboot the servers and wait for them to be discovered. Discovered nodes will be counted in top right corner of the Fuel dashboard.


    Now you can add UNALLOCATED NODES to the setup.
    First you can add Controller, Storage, and then Compute nodes.


    1. Access the Nodes tab in your environment.
    2. Click on the cogwheel to the right of a node:
    3. Verify that the driver is eth_ipoib and the bus information is an ib interface.


    Add Controller Nodes

    1. Click Add Node.
    2. Identify three controller node. Use the last four Hexadecimal digits of its MAC address of interface connected to the Admin (PXE) network. Assign the node's role as a Controller node.
    3. Click the Apply Changes button.

    Add Storage Node

    1. Click Add Node.
    2. Identify your storage node. Use the last 4 Hexadecimal digits of its MAC address of interface connected to Admin (PXE) network. In our example this is the only Supermicro server, so identification by vendor is easy. Select this node to be a Storage - Cinder node.
    3. Click the Apply Changes button.

    Add Compute Nodes

    1. Click Add Node.
    2. Select all the nodes that are left and assign them the Compute role.
    3. Click Apply Changes.


    Configure Interfaces

    In this step, each network must be mapped to a physical interface for each node. You can choose and configure multiple nodes in parallel.

    In case of hardware differences between selected nodes (like the number of network ports), bulk configuration is not allowed. If you do a bulk configuration, the Configure Interfaces button displays an error icon.


    Select all Controller and Compute nodes (6 nodes in this example) in parallel. The 7th node (Supermicro storage node) will be configured separately


    In this example, we set the Admin (PXE) network to eno1 and the Public network to eno2 interfaces.

    The Storage, Private and Management networks should run on the ConnectX-4 adapter's 100GbE port.

    This is an example:

    Click Back To Node List and perform network configuration for Storage Node by analogy.

    Note: In some cases the speed of the Mellanox interface is shown incorrectly, for example as 34.5 Gb/s in above image.


    Configure Disks

    Note: Use the defaults for the Controller and Compute nodes unless changes are required. For the Storage node it is recommended that you allocate only high-performing RAID as the Cinder storage. The small disk must allocated to the Base System.
    1. Select the Storage node.
    2. Press the Disk Configuration button.
    3. Click on the sda disk bar, use the slider lever to set Cinder allowed space to 0 MB and make the Base System occupy the entire drive.
    4. Click Apply.

    Networks Tab


    Section Node Network Group - Default


    In our example nodes the Public IP range is (seven used for physical servers, one is reserved for HA and one is reserved for virtual router).

    See Network Allocation section at the beginning of this post for details.

    The rest of public IP range will be used for Floating IP in Neutron L3 section below.
    In our example, Public network does not use VLAN. If you use VLAN for the Public network you should check Use VLAN tagging and specify the proper VLAN ID.





    In this example, we select VLAN 3 for the storage network. The CIDR is left untouched.



    In this example, we select VLAN 2 for the management network. The CIDR is left untouched.

    Section Settings

    Neutron L2

    In this example, we set the VLAN range to 4-100. It should be aligned with the switch VLAN configuration (above).
    The base MAC address is left unchanged.


    Neutron L3

    Floating Network Parameters: The floating IP range is a continuation of our Public IP range. In our deployment we use the - IP range.

    Internal Network: Leave CIDR and Gateway unchanged.

    Name servers: Leave DNS servers unchanged.


    We assign Public IP to all nodes. Make sure Assign public network to all nodes is checked.

    We use Neutron L3 HA option.

    For deployment we use and DNS servers as well.

    Save Your Configuration

    Click Save Settings at the bottom of page.


    Verify Networks

    Click Verify Networks.
    You should see the following message: Verification Failed. You will receive errors for Tenant (Neutron L2) VLANs as below. The cause of the error messages is missing VLANs/PKeys at this stage. Those VLANs/PKeys will be provisioned later by NEO for relevant ports only.

    Note: If your public network runs a DHCP server, you can experience a verification failure. If the range selected for the cloud above is not overlapping with DHCP pool, you can ignore this message. If overlap exists, please fix it.



    Click the Deploy Changes button and view the installation progress at the nodes tab and view logs.

    The OS installation will start.


    When the OS installation is finished, the OPSTK installation on 1st controller begin.

    Then OPSTK will be installed on rest of controllers, and afterwards on Compute and Storage nodes.


    Health Check

    1. Click on the Health Check tab.
    2. Check the Select All checkbox.
    3. Click Run Tests.

        All tests should pass. Otherwise, check the log file for troubleshooting information.


    You can now safely use the cloud.

    Click the Dashboard tab and then the Horizon link to enter Openstack dashboard.


    Default Network Re-Creation

    Warning: Default network is not operational because it was created without using NEO and is not provisioned properly.

    Workaround: The default network should be removed from Openstack and created from scratch.

    This can be done either from WEB UI or from CLI.


    The following example shows the steps for using the CLI.

    1. Login to Fuel node.
    2. Display the list of available controller nodes:
      [root@fuel ~]# fuel node | grep -i true |grep -i controller
      2  | ready  | Untitled (43:f8) | 1       | | a0:d3:c1:01:43:f8 | controller |               | True   | 1      
      3  | ready  | Untitled (44:58) | 1       | | a0:d3:c1:01:44:58 | controller |               | True   | 1      
      1  | ready  | Untitled (43:68) | 1       | | a0:d3:c1:01:43:68 | controller |               | True   | 1
    3. SSH to any of the controller nodes and source openrc file.
      The openrc file contains all variables required for correct use of Openstack CLI.
      [root@fuel ~]# ssh node-1
      Warning: Permanently added 'node-1' (ECDSA) to the list of known hosts.
      Welcome to Ubuntu 14.04.5 LTS (GNU/Linux 3.13.0-93-generic x86_64)

      * Documentation:
      root@node-1:~# source openrc
    4. Use the commands described below to re-create the default tenant network.

      ## Removing router interface
      neutron router-interface-delete router04 `neutron router-port-list router04|grep '192.168.111' | awk '{print $8}' | cut -d'"' -f 2`

      ## Removing all ports of the default network

      for i in `neutron port-list | grep '192.168.111' | awk '{print $2}'`; do neutron port-delete $i; done

      ## Removing default subnet

      neutron subnet-delete `neutron subnet-list | grep '192.168.111' | awk '{print $2}'`

      ## Removing default network

      neutron net-delete `neutron net-list|grep 'admin_internal_net' | awk '{print $2}'`

      ## Creating default network

      neutron net-create admin_internal_net --provider:network-type vlan

      ## Creating subnet for default network

      neutron subnet-create admin_internal_net --name admin_internal_net__subnet --enable-dhcp --dns-nameserver

      ## Linking router to external network

      neutron router-gateway-set router04 `neutron net-list | grep float|awk '{print $2}'`

      ## Linking the route to the subnet

      neutron router-interface-add router04 admin_internal_net__subnet


    Start the SR-IOV VM

    In Liberty each VM can start either with standard the Para-Virtual or SR-IOV network port. By default the Para-Virtual OVS-connected port is used. You need to request vnic_type direct explicitly in order to assign SR-IOV NIC to the VM. First you need to create SR-IOV Neutron port and then spawn the VM with the port attached. Since this feature is not available in the UI, we will describe how to start the VM from the SR-IOV network port using the CLI.

    Note: The VM must have the Mellanox NIC driver installed in order to have a working network.

    Note:  The VM should have the ib0 pre-configured interface in order to have connectivity after deployment.

    Using Mellanox Scripts for SR-IOV Deployment

    Ensure the VM image <image_name> has the Mellanox driver installed.
    We recommend that you use the most recent Mellanox OFED version. See Mellanox OpenFabrics Enterprise Distribution for Linux (MLNX_OFED) for more information.

      1. Upload the qcow2 image to Cloud Controller node.
      2. Log in to the cloud Controller Node and source openrc.
      3. Upload the qcow2 image to check.
        # upload_sriov_image -f <image_name>
      4. Check the list of images in the system to make sure that new image is present.
        # nova image-list
        The new image should have name like mellanox-<image_name>.
      5. Check the list of flavors that can be used.
        # nova flavor-list
        Note: Keep in mind that if you use a flavor with too low parameters, especially disk size, VM might fail to deploy.
      6. Start SR-IOV VM using the uploaded image and selected flavor.
        # start_sriov_vm -f <flavor_name>

    Starting SR-IOV VM Manually

      1. Login to the cloud Controller Node and source openrc.
      2. Create the SR-IOV enabled neutron port first.

        # port_id=`neutron port-create $net_id --name sriov_port --binding:vnic_type direct | grep "\ id\ " | awk '{print $4}'`
        where $net_id is ID or Name of the network You want to use

      3. Then start the new VM and bind it to the port you just created.

        # nova boot --flavor <flavor_name> --image <image_name> --nic port-id=$port_id <vm_name>

        where $port_id is ID of our just created SR-IOV port


    Checking SR-IOV

    To make sure that SR-IOV works, log in to the VM's console and run the lspci command.
    You should see the Mellanox VF in the output.

    # lspci | grep -i mellanox

    00:04.0 Network controller [0207]: Mellanox Technologies MT27621 Family



    Prepare the Linux VM Image for CloudX

    In order to have network and RoCE support on the VM, MLNX_OFED (use the most current version) install it on the VM.

    MLNX_OFED can be downloaded from

    (In case of CentOS/RHEL OS, you can use virt-manager to open the existing VM image and perform the MLNX_OFED installation).



    Known Issues:

    Issue #DescriptionWorkaround
    1Configuring more than 16 VFs is not supported by default.

    To have more vNICs available, contact Mellanox Support.
    2Max 31 VF is supported.Please do not use more.
    3Verify network can cause error message if public network has running DHCP server.If the range selected for the cloud above is not overlapping with DHCP pool, ignore this message. If an overlap exists, please fix it.
    4Verify network before installation show error message about missing VLANs if cloud is using Mellanox NEO
    Please ignore this message, missing VLANs/PKs will be provisioned by NEO later.
    5Launching more than 3 VMs at once may cause fail of those VMs.

    Delete all such VMs.

    6Migration of SR-IOV based VM is not working with nova migrate command.N/A
    7In some cases the speed of the Mellanox interface can be shown as less then 100Gb/s.N/A

    You can't create more than 125 virtual networks.

    This is Infiniband limitation with no workaround available.

    Amount of PKeys is limited to 128. We use PKeys 2 and 3 for cloud Management and Storage networks. Additional PKey is utilized for admin_internal_net network.