CloudX with MOS 9.0 over Mellanox ConnectX-4 Adapters (CloudX with MOS 8 complimentary post)

Version 5

    This is a short post that covers difference in setting up and configuring Mirantis Openstack 8.0 (Liberty) and Mirantis Openstack 9 (Mitaka).

    Warning: This product is in status of BETA. It is NOT officially supported. Please use it with caution and on your own risk.

    Before reading this post, make sure you are familiar with Mirantis Openstack 9.0 and CloudX with MOS 8.0 over Mellanox ConnectX-4 Adapters (Ethernet Network + BOND + SR-IOV + VLAN Segmentation) installation procedures.


    In this example we are preparing to set up cloud, which can be scaled in the future.

    Main highlights are:

    • Cloud runs with a flat network using two Mellanox SN2700 switches for redundancy.
    • Cloud can be scaled up to 30 nodes.
    • Cloud runs with ConnectX-4 (100Gbps) NICs.
      • Each Mellanox NIC is wired to both Mellanox switches.
      • BOND works in mode Active-Backup.
    • Cloud runs with three controller nodes for HA mode.
    • All cloud nodes are connected to Admin (PXE), Public, Private, Management and Storage networks.
    • Fuel master is running as VM on the Deployment node.

    Note:You can use more controllers, but the number should always be an odd number.

    Note:The server’s IPMI wiring and configuration are out of the scope of this post.

    Setup Diagram




    Note:Besides the Deployment node, all nodes should be connected to all five networks.

    Note:Server’s IPMI and the switches management interfaces wiring and configuration are out of scope. Ensure that there is management access (SSH) to Mellanox switch SN2700 so that you can perform the configuration.

    Install the Fuel Master from an ISO Image

    2. Choose option 1 and press the TAB button to edit default options:

      1. Remove the default gateway (
      2. Change the DNS to (or any IP convenient for you).
      3. Add the following command to the end: showmenu=yes



    8. Once the Fuel installation is done, you will see Fuel access details both for SSH and HTTP.


    Configure the Fuel Master VM SSH server to allow connections from the Public network.

    By default Fuel will accept SSH connections from Admin(PXE) network only.

    Follow the steps below to allow connections from Public Network:

    By default Fuel will accept SSH connections from Admin(PXE) network only.


    Use virt-manager to access Fuel Master VM console and follow the steps below to allow connections from Public Network:

    1. Update SSH config:
      1. Edit sshd_config:
        # vi /etc/ssh/sshd_config
      2. Find and comment line:
      3. Restart sshd:
        # service sshd restart
    2. Open SSH port in firewall:
      1. Edit iptables file
        # vi /etc/sysconfig/iptables
      2. Find firewall rule
        -A INPUT -s -p tcp -m multiport --dports 22 -m comment --comment "010 ssh" -m state --state NEW -j ACCEPT
        and add below:
        -A INPUT -i eth1 -p tcp -m multiport --dports 22 -m comment --comment "Allow SSH from Public" -m state --state NEW -j ACCEPT
      3. Apply updated rules
        # iptables-restore /etc/sysconfig/iptables


    Install the Mellanox Plugin

    The Mellanox plugin:

    • Configures support for Mellanox ConnectX-4 network adapters
    • Enables high-performance SR-IOV compute traffic networking
    • Brings support for block storage networking for iSER (iSCSI), which reduces CPU overhead, boosts throughput, reduces latency, and enables network traffic to bypass the software switch layer.


    Follow the steps below to install the plugin:

    1. Login to the Fuel Master. Download the plugin rpm file for MOS 9.0 from Mellanox page and store it on your Fuel Master server.
    2. To copy the plugin on the already-installed Fuel Master node, use the scp command as follows:
      # scp mellanox-plugin-3.3-3.3.0-1.noarch.rpm root@<Fuel_Master_ip>:/tmp
    3. Install plugin from tmp directory:

      # cd /tmp
      # fuel plugins --install mellanox-plugin-3.3-3.3.0-1.noarch.rpm

      Note: The Mellanox plugin replaces the current bootstrap image.

    4. Verify that the plugin was successfully installed. It should be displayed when running the fuel plugins command.
      # fuel plugins
    5. Create a new bootstrap image by running create_mellanox_bootstrap --link_type {eth,ib,current} --max_num_vfs={amount_VFs}.
      For Eth setup run:
      [root@fuel ~]# create_mellanox_bootstrap --link_type eth --max_num_vfs=16
      Try to build image with data:
      certs: null
      container: {format: tar.gz, meta_file: metadata.yaml}
      . . .
      . . .
      . . .
      Bootstrap image f790e9f8-5bc5-4e61-9935-0640f2eed949 has been activated

      Sixteen is the default number of Virtual Functions. You can use the --max_num_vfs parameter to change the default value.
      This amount will be burned in Mellanox ConnectX-4 firmware.
      Use the --max_num_vfs parameter to specify the number of Virtual Functions available for each NIC.
      For example you can use:
      [root@fuel ~]# create_mellanox_bootstrap --link_type eth --max_num_vfs 16
      Note: Maximum supported value is 31. Please consult with Mellanox support if you would like to use more than 16 Virtual Functions (VF)s.
    6. Reboot all nodes, including already-discovered nodes.
      To reboot already-discovered nodes by running the following command:
      # reboot_bootstrap_nodes -a


    Configure the OpenStack Environment

    Go to the Settings tab, General section

    Find line Provision in the bottom of the form

    Add package systemd-shim to Initial packages field

    Go to Settings tab, section Other

    Mark checkbox "Propagate task based deployment" before you will procceed