CloudX with MOS 8.0 over Mellanox ConnectX-4 Adapters (Ethernet Network + BOND + SR-IOV + VLAN Segmentation)

Version 15

    This post shows how to set up and configure Mirantis Fuel 8 (OpenStack Liberty based on Ubuntu 14.04) to support Mellanox ConnectX-4 adapters with bonding. This procedure enables SR-IOV mode for the Virtual Machines (VMs) on the compute nodes and iSER transport mode for the storage nodes.



    Before reading this post, make sure you are familiar with Mirantis Fuel 8.0 installation procedures.


    In this example we are preparing to set up cloud, which can be scaled in the future.

    Main highlights are:

    • Cloud runs with a flat network using two Mellanox SN2700 switches for redundancy.
    • Cloud can be scaled up to 30 nodes.
    • Cloud runs with ConnectX-4 (100Gbps) NICs.
      • Each Mellanox NIC is wired to both Mellanox switches.
      • BOND works in mode Active-Backup.
    • Cloud runs with three controller nodes for HA mode.
    • All cloud nodes are connected to Admin (PXE), Public, Private, Management and Storage networks.
    • Fuel master is running as VM on the Deployment node.

    Note: You can use more controllers, but the number should always be an odd number.

    Note: The server’s IPMI wiring and configuration are out of the scope of this post.


    Setup Diagram




    Note: Besides the Deployment node, all nodes should be connected to all five networks.

    Note: Server’s IPMI and the switches management interfaces wiring and configuration are out of scope. Ensure that there is management access (SSH) to Mellanox switch SN2700 so that you can perform the configuration.

    Setup Hardware Requirements (Example)

    Deployment Node1 x

    Select a non-high performing server to run Fuel VM with the following:

    CPU: Intel E5-26xx or later model

    HD: 250 GB or larger

    RAM: 32 GB or more


    • 2 x 1Gb
    • Mellanox ConnectX-4 VPI dual port (MCX456A-ECAT) or single port (MCX455A-ECAT) NIC

    Cloud Controllers and Compute Servers:

    • 3 x Controllers
    • 3 x Computes
    6 x

    Strong servers to run Cloud control and Tenant’s VM workload that meet the following:

    CPU: Intel E5-26xx or later model

    HD: 450 GB or larger

    RAM: 128 GB or more


    • Two 1GbE ports
    • A Mellanox ConnectX-4 VPI dual port (MCX456A-ECAT) or single port (MCX455A-ECAT) NIC
    Cloud Storage Server1 x

    A high IO performance strong server to act as Cinder Volume backend featuring:

    CPU: Intel E5-26xx or later model


    • OS: 100 GB or larger
    • Cinder Volume: SSD drives configured in RAID-10 for best performance

    RAM: 64 GB or moreNICs:

    • Two 1GbE ports
    • A Mellanox ConnectX-4 VPI dual port (MCX456A-ECAT) or single port (MCX455A-ECAT) NIC
    Admin (PXE) Switch1 x1Gb switch to access Admin (PXE) network
    Public Switch1 x1Gb switch with VLANs configured to access public network
    Ethernet Switch2 xMellanox SN2700 32 ports 100Gbps Ethernet switch for Private, Management and Storage networks.

    8 x

    8 x

    16 x

    1Gb CAT-6e Public network

    1Gb CAT-6e Admin (PXE) network

    100GbE copper cables up to 2m long (MCP1600-EXXX)


    Note: Intel Virtualization technologies and SR-IOV must be enabled in BIOS.


    Storage Server RAID Setup

    • Two SSD drives in bays 0-1 configured in RAID-1 (Mirror) are used for the OS.

    • Twenty-two SSD drives in bays 3-24 configured in RAID-10 are used as a Cinder volume and will be configured on the RAID drive.



    Network Physical Setup

    1. Connect all nodes to the Admin (PXE) 1GbE switch (preferably through the eth0 interface on board).
      We recommend that you record the MAC address of the Controller and Storage servers to make Cloud installation easier (see Controller Node section below in Nodes tab).
      Note: All cloud servers should be configured to run PXE boot over the Admin (PXE) network.
    2. Connect all nodes to the Public 1GbE switch (preferably through the eth1 interface on board).
    3. Connect Port #1 of the ConnectX-4 to first and Port #2 to the second of TOR SN2700 Ethernet switches (Private, Management, and Storage networks).
      Note: The interface names (eth0, eth1, p2p1, etc.) might vary between servers from different vendors.

    Rack Setup Example

    Deployment Node

    Compute and Controller Nodes

    Storage Node
    The configuration is the same as it is for the Compute and Controller nodes.


    Network Switch Configuration

    1. Configure Inter Peer Link (IPL) between Mellanox switches.
      Note: Refer to the MLNX-OS User Manual to become familiar with switch software (located at
      Note: Before starting to use of the Mellanox switch, we recommend that you upgrade the switch to the latest MLNX-OS version.

      1. Initiate switches.
        On both run the following:
        switch > enable
        switch # configure terminal
        switch  [standalone: master] (config) # lacp
        switch  [standalone: master] (config) # lldp
        switch  [standalone: master] (config) # no spanning-tree
        switch  [standalone: master] (config) # ip routing
        switch  [standalone: master] (config) # protocol mlag
        switch  [standalone: master] (config) # dcb priority-flow-control enable force
      2. Now we need to configure IPL between  switches (sw-1 and sw-2).
        In this example VLAN 4001 is used for the IPL.
        On both switches run the following:
        switch  [standalone: master] (config) # interface port-channel 1
        switch  [standalone: master] (config interface port-channel 1) # exit
        switch  [standalone: master] (config) # interface ethernet 1/31 channel-group 1 mode active
        switch  [standalone: master] (config) # interface ethernet 1/32 channel-group 1 mode active
        switch  [standalone: master] (config) #
        switch  [standalone: master] (config) # vlan 4001
        switch  [standalone: master] (config vlan 4001) # exit
        switch  [standalone: master] (config) #
        switch  [standalone: master] (config) # interface vlan 4001
        switch  [standalone: master] (config interface vlan 4001) # exit
        switch  [standalone: master] (config) #
        switch  [standalone: master] (config) # interface port-channel 1 ipl 1
        switch  [standalone: master] (config) # interface port-channel 1 dcb priority-flow-control mode on force
      3. Assign IP addresses to VLAN interfaces and configure peers:
        On sw-1 run:
        switch  [standalone: master] (config) # interface vlan 4001
        switch  [standalone: master] (config interface vlan 4001) # ip address
        switch  [standalone: master] (config interface vlan 4001) # ipl 1 peer-address
        switch  [standalone: master] (config interface vlan 4001) # exit

        On sw-2 run:
        switch  [standalone: master] (config) # interface vlan 4001
        switch  [standalone: master] (config interface vlan 4001) # ip address
        switch  [standalone: master] (config interface vlan 4001) # ipl 1 peer-address
        switch  [standalone: master] (config interface vlan 4001) # exit

      4. Now we need to assign Virtual IP, MAC and domain name:
        On both run:
        switch  [standalone: master] (config) # mlag-vip <DOMAIN_NAME> ip <VIRTUAL_IP> /24 force
        switch  [standalone: master] (config) # mlag system-mac 00:00:5E:00:01:5D
        switch  [standalone: master] (config) # no mlag shutdown
        Note: IP addresses as well as MAC addresses are just examples. Please replace them with suitable for you.
    2. Configure the required VLANs and enable flow control on the Ethernet switch ports.
      All related VLANs should be enabled on the 100GbE switches (Private, Management, Storage networks).
      In this example VLANs 1-4000 are configured:
      * VLAN 1 is the default.
      * VLANs 2 and 3 are used for Management and Storage networks respectively.
      * VLANs 4-4000 are designated as tenant networks.
      Warning: VLANs used for tenant networks should not overlap VLAN used for IPL (4001 in this example).
      1. On Mellanox switches, use the command flow below to enable VLANs and enable flow control on the Ethernet switch ports.
        On both switches run:
        switch (config) # vlan 1-4000
        switch (config vlan 1-4000) # exit
        switch (config) # interface ethernet 1/1 switchport mode hybrid
        switch (config) # interface ethernet 1/1 switchport hybrid allowed-vlan all
        switch (config) # interface ethernet 1/2 switchport mode hybrid
        switch (config) # interface ethernet 1/2 switchport hybrid allowed-vlan all

        switch (config) # interface ethernet 1/30 switchport mode hybrid
        switch (config) # interface ethernet 1/30 switchport hybrid allowed-vlan all
      2. Flow control is required when running iSER (RDMA over RoCE - Ethernet).
        On Mellanox switches, run the following command to enable flow control on the switches (on all ports connected to nodes in this example):
        switch (config) # interface ethernet 1/1-1/30 flowcontrol receive on force
        switch (config) # interface ethernet 1/1-1/30 flowcontrol send on force
      3. To save the configuration (permanently), run:
        switch (config) # configuration write


    Networks Allocation (Example)

    The example in this post is based on the network allocation defined in this table:

    Admin (PXE) network is used to provision and manage cloud nodes by the Fuel Master. The network is enclosed within a 1Gb switch and has no routing outside. is the default Admin (PXE) subnet and we use it with no changes.
    Management192.168.0.0/24N/AThis is the Cloud Management network. The network uses VLAN 2 in SN2700 over a 100Gb interconnect. is the default Management subnet and we use it with no changes.
    Storage192.168.1.0/24N/AThis network is used to provide storage services. The network uses VLAN 3 in SN2700 over a 100Gb interconnect. is the default Storage subnet and we use it with no changes.
    Public and Neutron L310.7.208.0/2410.7.208.1

    A Public network is used to connect Cloud nodes to an external network.

    Neutron L3 is used to provide Floating IP for tenant VMs.

    Both networks are represented by IP ranges within the same subnet with routing to external networks.


    All Cloud nodes are assigned Public IP addresses. In addition, you must allocate two more Public IP addresses.

    • One IP required for HA functionality.
    • A virtual router requires additional Public IP addresses.

    We do not use a virtual router in our deployment but still need to reserve Public IP address for it. So the Public Network range is the number of cloud nodes + 2. For our example with 7 Cloud nodes we need 9 IPs in the Public network range.

    Note: Consider a larger range if you are planning to add more servers to the cloud later.

    In our build we use a >> IP range for both Public and Neutron L3. IP allocations are as follows:

    • Deployment node:
    • Fuel Master IP:
    • Public Range: >> (7 used for physical servers, 1 reserved for HA and 1 reserved for virtual router)
    • Neutron L3 Range: >> (used for Floating IP pool)




    The following scheme illustrates the public IP range allocation for the Fuel VM with all setup nodes and the Neutron L3 floating IP range.


    Install the Deployment Server

    In our setup we install the CentOS release 7.3 64-bit distribution. We use the CentOS-7-x86_64-Minimal-1511.iso image and installed the minimal configuration. We will install all missing packages later.

    Two 1Gb interfaces are connected Admin (PXE) and Public:

    • em1 (first interface of integrated NIC) is connected to Admin (PXE) and configured statically.

      The configuration will not actually be used, but will save time on bridge creation later.

        • IP:

        • Netmask:

        • Gateway: N/A

    • em2 (second interface of integrated NIC) is connected to Public and configured statically:

        • IP:

        • Netmask:

        • Gateway:



    Configure the Deployment Node for Running the Fuel VM


    Login to the Deployment node by SSH or locally and perform actions listed below:

    1. Disable the Network Manager.
      # sudo systemctl stop NetworkManager.service
      # sudo systemctl disable NetworkManager.service
    2. Install packages required for virtualization.
      # sudo yum install qemu-kvm libvirt libvirt-python libguestfs-tools virt-install virt-manager
    3. Install packages required for x-server.
      # sudo yum install xorg-x11-xauth xorg-x11-server-utils xclock
    4. Reboot the Deployment server.


    Create and Configure the new VM To Run the Fuel Master Installation

    1. Start the VM.
      # virt-manager
    2. Create a new VM using the virt-manager wizard.
    3. During the creation process provide VM with four cores, 8GB of RAM and 200GB disk.
      Note: For details see the Fuel Master node hardware requirements section in Mirantis OpenStack Planning Guide — Mirantis OpenStack v8.0 | Documentation.
    4. Mount the Fuel installation disk to VM virtual CD-ROM device.
    5. Configure the network so the Fuel VM has two NICs connected to Admin (PXE) and Public networks.
      1. Use virt-manager to create bridges.

        1. br-em1, the Admin (PXE) network bridge used to connect Fuel VM's eth0 to Admin (PXE) network.

        2. br-em2, the Public network bridge used to connect Fuel VM's eth1 to Public network.

      2. Connect the Fuel VM eth0 interface to br-em1.

      3. Add to the Fuel VM eth1 network interface and connect it to br-em2.

        Note: You can define any other names for bridges. In this example names were defined them to match names of the physical interfaces that are connected to the Deployment node.
    6. Save settings and start VM.


    Install the Fuel Master from an ISO Image

    Note: Avoid starting the other nodes except for the Fuel Master until the Mellanox plugin is installed.

    1. Boot the Fuel Master Server from the ISO image as a virtual DVD (click here to download ISO image).
    2. Choose option 1 and press the TAB button to edit default options:
      1. Remove the default gateway (
      2. Change the DNS to (the Fuel VM IP).
      3. Add the following command to the end: showmenu=yes
        The tuned boot parameters should look like this:

        Note: Do not try to change eth0 to another interface or the deployment might fail.
    3. Fuel VM will reboot itself after the initial installation is completed and the Fuel menu will appear.
      Note: Ensure that the VM will start from the Local Disk and not CD-ROM. Otherwise you will restart the installation from beginning.
    4. Begin the network setup:
      1. Configure eth0 as the PXE (Admin) network interface.

        Ensure the default Gateway entry is empty for the interface. The network is enclosed within the switch and has no routing outside.

        Select Apply.

      2. Configure eth1 as the Public network interface.

        The interface is routable to LAN/internet and will be used to access the server. Configure the static IP address, netmask and default gateway on the public network interface.

        Select Apply.

    5. Set the PXE Setup.
      The PXE network is enclosed within the switch. Use the default settings.
      Press the Check button to ensure no errors are found.
    6. Set the Time Sync.
      1. Choose the Time Sync option on the left-hand menu.
      2. Configure the NTP server entries suitable for your infrastructure.
      3. Press Check to verify settings.
    7. Proceed with the installation.
      Navigate to Quit Setup and select Save and Quit.
    8. Once the Fuel installation is done, you will see Fuel access details both for SSH and HTTP.

      Configure the Fuel Master VM SSH server to allow connections from the Public network.
      By default Fuel will accept SSH connections from Admin(PXE) network only.
      Follow the steps below to allow connections from Public Network:
      1. Use virt-manager to access Fuel Master VM console.
      2. Edit sshd_config:
        # vi /etc/ssh/sshd_config
      3. Find and comment line:
      4. Restart sshd:
        # service sshd restart
    9. Access Fuel using one of the following:
      • Web UI by (use admin/admin as user/password)
      • SSH by connecting (use root/r00tme as user/password)


    Maintenance Updates

    It is highly recommended that you install Mirantis Fuel maintenance updates before proceeding with cloud deployment.

    Please read Mirantis Openstack v8.0 — Maintenance Updates section to learn how to apply updates.


    Install the Mellanox Plugin

    The Mellanox plugin:

    • Configures support for Mellanox ConnectX-4 network adapters
    • Enables high-performance SR-IOV compute traffic networking
    • Brings support for block storage networking for iSER (iSCSI), which reduces CPU overhead, boosts throughput, reduces latency, and enables network traffic to bypass the software switch layer.


    Follow the steps below to install the plugin. For the complete instructions, please refer to: HowTo Install Mellanox OpenStack Plugin for Mirantis Fuel 8.0).

    1. Login to the Fuel Master. Download the plugin rpm file for MOS8.0 from Mellanox page and store it on your Fuel Master server.
    2. To copy the plugin on the already-installed Fuel Master node, use the scp command as follows:
      # scp mellanox-plugin-3.2-3.2.1-1.noarch.rpm root@<Fuel_Master_ip>:/tmp
    3. Install plugin from tmp directory:
      # cd /tmp
      # fuel plugins --install mellanox-plugin-3.2-3.2.1-1.noarch.rpm
      Note: The Mellanox plugin replaces the current bootstrap image.
    4. Verify that the plugin was successfully installed. It should be displayed when running the fuel plugins command.
    5. Create a new bootstrap image by running create_mellanox_bootstrap --link_type {eth,ib,current} --max_num_vfs={amount_VFs}.
      For Eth setup run:
      [root@fuel ~]# create_mellanox_bootstrap --link_type eth --max_num_vfs=16
      Try to build image with data:
      certs: null
      container: {format: tar.gz, meta_file: metadata.yaml}
      . . .
      . . .
      . . .
      Bootstrap image f790e9f8-5bc5-4e61-9935-0640f2eed949 has been activated

      Sixteen is the default number of Virtual Functions. You can use the --max_num_vfs parameter to change the default value.
      This amount will be burned in Mellanox ConnectX-4 firmware.
      Use the --max_num_vfs parameter to specify the number of Virtual Functions available for each NIC.
      For example you can use:
      [root@fuel ~]# create_mellanox_bootstrap --link_type eth --max_num_vfs 16
      Note: Maximum supported value is 31. Please consult with Mellanox support if you would like to use more than 16 Virtual Functions (VF)s.
    6. Reboot all nodes, including already-discovered nodes.
      To reboot already-discovered nodes by running the following command:
      # reboot_bootstrap_nodes -a



    Enabling DHCP Broadcast

    In this example we are using bond in Active-Backup mode. As we do not know which of the interfaces will become active, VM will not obtain the IP address from the DHCP-agent if the second interface is used as primary.

    As a workaround we should enable DHCP broadcast.

    1. Edit the config file:
      vi /etc/puppet/liberty-8.0/modules/neutron/manifests/agents/dhcp.pp
    2. Find the line:
      $dhcp_broadcast_reply     = false
    3. Change 'false' to 'true'
      $dhcp_broadcast_reply     = true

    Now, DHCP broadcast will be enabled on all controller nodes automatically after deployment.


    Create a new OpenStack Environment

    Open in WEB browser (for example: and log into Fuel environment using admin/admin as the username and password.


    1. Open a new environment in the Fuel dashboard. A configuration wizard will start.
    2. Configure the new environment wizard as follows:
      • Name and Release
        • Name: SR-IOV
        • Release: Liberty on Ubuntu 14.04
      • Compute
        • QEMU-KVM
      • Network
        • Neutron with VLAN segmentation
      • Storage Backend
        • Block storage: LVM
      • Additional Services
        • None
      • Finish
        • Click the Create button.
    3. Click on the newly-created environment and proceed with the environment configuration.

    Configure the OpenStack Environment


    Settings Tab

    1. Specify that KVM is the hypervisor type.
      KVM is required to enable Mellanox Openstack features.
      Open the Settings tab, select the Compute section, and then choose KVM as the hypervisor type.
      Save your settings.

    2. In General section add intel_iommu=on to kernel parameters.

      Save your settings.
    3. Enable the desired Mellanox OpenStack features.
      1. Open the Other section.
      2. Select relevant Mellanox plugin versions if you have multiple versions installed.
      3. Enable SR-IOV support.
        • Check the SR-IOV direct port creation in private VLAN networks (Neutron) option.
        • Set the desired number of Virtual NICs.

          Note: Relevant for VLAN segmentation only
          Note: The number of virtual NICs is amount of VFs that will be available on the Compute node. It should be less or equal to value used during bootstrap creation (default is 16).
          Note: One VF will be utilized for iSER storage transport if you choose to use iSER. In this case you will get one VF less for Virtual Machines.
      4. Enable the iSCSI Extension over RDMA (iSER) protocol for volumes (Cinder).

        By enabling this feature you will use the iSER  block storage transport instead or ISCSI. iSER provides improved latency, better bandwidth and reduced CPU overhead.
      5. Save your settings.

      Nodes Tab


      Server Discovery by Fuel

      This section assigns Cloud roles to servers. Servers should be discovered by Fuel. So you must ensure the servers are configured for PXE boot over the Admin (PXE) network. When done, reboot the servers and wait for them to be discovered. Discovered nodes will be listed in top right corner of the Fuel dashboard.

      Now you can add UNALLOCATED NODES to the setup.
      First you can add Controller, Storage, and then Compute nodes. A description of how to select each follows.


      Add Controller Nodes

      1. Click Add Node.
      2. Identify three controller nodes. Use the last four hexadecimal digits of its MAC address for the interface connected to Admin (PXE) network. Assign the node's role as a Controller node.
      3. Click the Apply Changes button.

      Add the Storage Node

      1. Click Add Node.
      2. Identify your storage node. Use the last four hexadecimal digits of its MAC address for the interface connected to the Admin (PXE) network. In our example this is the only Supermicro server, so identification by vendor is easy. Specify this node as a Storage - Cinder node.
      3. Click Apply Changes.

      Add Compute Nodes

      1. Click Add Node.
      2. Select all the nodes that are remaining and specify them as having Compute roles.
      3. Click Apply Changes.


      Configure Interfaces

      In this step, each network must be mapped to a physical interface for each node. You can choose and configure multiple nodes in parallel.

      In case of HW differences between selected nodes (like the number of network ports), bulk configuration is not allowed. If you perform a bulk configuration, the Configure Interfaces button displays an error icon.


      The example below shows you how to configure six nodes in parallel. The seventh node (the Supermicro storage node) is configured separately.


      In this example, we set the Admin (PXE) network to eno1 and the Public network to eno2.

      The Storage, Private, and Management networks should run on the ConnectX-4 adapters 100GbE port.

      In order to have redundancy, both ports of ConnectX-4 adapters should be joined in the bond interface mode for Active Backup.

      Click Back To Node List and perform network configuration for the Storage Node.

      Note: In some cases speed of the Mellanox interface can be shown incorrectly, for example as 35Gbps.


      Configure Disks

      Note: There is no need to change the defaults for the Controller and Compute nodes unless changes are required. For the Storage node we recommend that you allocate only high performing RAID for Cinder storage. The small disk will be allocated to Base System.

      1. Select the Storage node.
      2. Press the Disk Configuration button.
      3. Click on the sda disk bar, specify that Cinder be allowed 0 MB space and make the Base System occupy the entire drive.
      4. Click Apply.


      Networks Tab



      Section Node Network Group settings are displayed by default.



      In our example nodes Public IP range is (seven are used for physical servers, one is reserved for HA, and one reserved for virtual router).

      See the Network Allocation section at the beginning of this document for details.

      The rest of the public IP range is used for Floating IP in Neutron L3 section below.
      In our example, the Public network does not use VLAN. If you use VLAN for the Public network, you should check Use VLAN tagging and specify the proper VLAN ID.





      In this example, we select VLAN 3 for the Storage network. The CIDR is unchanged.


      In this example, we select VLAN 2 for the Management network. The CIDR is unchanged.


      Section Settings


      Neutron L2

      In this example, we set the VLAN range to 4-4000. It should be aligned with the switch VLAN configuration (above).
      The Base MAC address is unchanged.


      Neutron L3
      • Floating Network Parameters: The floating IP range is a continuation of our Public IP range. In our deployment we use the - IP range.
      • Internal Network: Leave CIDR and Gateway unchanged.
      • Name servers: Leave DNS servers unchanged.


      We assign Public IP addresses to all nodes. Make sure that Assign public network to all nodes is checked.

      Use the Neutron L3 HA option.

      For deployment we use and DNS servers, as well.

      Save the Configuration

      Click Save Settings at the bottom of page.


      Verify Networks

      Click Verify Networks.
      You should see the following message: Verification succeeded. Your network is configured correctly. Otherwise, check the log file for troubleshooting.

      Note: If your public network runs a DHCP server, you can experience a verification failure. If the range selected for the cloud above is not overlapping with DHCP pool, you can ignore this message. If overlap exists, please fix it.


      Click the Deploy Changes button, view the installation progress at the Nodes tab, and view logs.

      The OS installation will start.


      When OS installation is finished, the OPSTK installation process on first controller starts.

      Then OPSTK will be installed on the rest of the controllers, and afterwards on the Compute and Storage nodes.


      The installation is completed.

      Health Check

      Click on the Health Check tab.

      1.    Check the Select All checkbox.

      2.    Click Run Tests.

          All tests should pass. Otherwise, check the log file for troubleshooting.

      You can now safely use the cloud.

      Click the dashboard link Horizon at the top of the page.


      Start the SR-IOV VM

      In Liberty each VM can start by using either the standard Para-Virtual or SR-IOV network port.

      By default the Para-Virtual, OVS-connected port will be used. You need to request vnic_type direct explicitly in order to assign SR-IOV NIC to the VM.

      First you need to create an SR-IOV Neutron port and then spawn VM with the port attached. In this example, we will show you how to start VM with SR-IOV network port using CLI, since this feature is not available in UI.

      Note: The VM must have Mellanox a NIC driver installed in order to get working network.


      Using Mellanox Script for SR-IOV Deployment

      Please, ensure the VM image <image_name> has Mellanox driver installed.
      We recommend to use most recent Mellanox OFED. See Mellanox OpenFabrics Enterprise Distribution for Linux (MLNX_OFED) for more information.

      1. Upload the qcow2 image to the Cloud Controller node.
      2. Login to the cloud Controller Node and source openrc.
      3. Upload the qcow2 image to Glance.
        # upload_sriov_image -f <image_name>
      4. Check the list of images in the system to make sure that the new image is present.
        # nova image-list
        The new image should have name like mellanox-<image_name>.
      5. Check list of options that can be used.
        # nova flavor-list
        Note: Please keep in mind, that if you will use flavor with too low parameters, especially disk size, VM may fail to deploy.
      6. Start SR-IOV VM using uploaded image and selected flavor
        # start_sriov_vm -f <flavor_name>


      Starting SR-IOV VM Manually

      Ensure the VM image <image_name> has Mellanox driver installed.
      We recommend that you use the most recent version of the Mellanox OFED driver. See Mellanox OpenFabrics Enterprise Distribution for Linux (MLNX_OFED) for more information.

      1. Login to cloud Controller node and source openrc.
      2. Create an SR-IOV-enabled neutron port first.
        # port_id=`neutron port-create $net_id --name sriov_port --binding:vnic_type direct | grep "\ id\ " | awk '{ print $4 }'`
        where $net_id is ID or Name of the network you want to use
      3. Then start the new VM and bind it to the newly-created port.
        # nova boot --flavor <flavor_name> --image <image_name> --nic port-id=$port_id <vm_name>
        where $port_id is ID of our just created SR-IOV port


      Checking SR-IOV

      To ensure that SR-IOV works, log in to the VM's console and run the lspci command.
      You should see the Mellanox VF in the output if SR-IOV works.

      # lspci | grep -i mellanox

      00:04.0 Network controller [0207]: Mellanox Technologies MT27621 Family

      Prepare Linux VM Image for CloudX

      In order to have network and RoCE support on the VM, MLNX_OFED remember to install the most recent version on the VM.

      MLNX_OFED can be downloaded from

      Note: If you are using CentOS/RHEL OS, you can use virt-manager to open the existing VM image and perform the MLNX_OFED installation.


      Assigning Usernames and Passwords

      • For the Fuel server Dashboard enter the user / password as: admin / admin
      • For the Fuel server SSH enter the user / password as: root / r00tme
      • To get controller node CLI permissions run:  # source /root/openrc

      Known Issues

      Issue #DescriptionWorkaround
      1Configuring more than 16 VFs is not supported by default.To have more vNICs available, please contact Mellanox Support.
      2Max 31 VF is supported.Do not exceed the 31 VFs.
      3Third-party adapters based on the Mellanox chipset may not have SR-IOV enabled by default.Apply to the device manufacturer for configuration instructions and for the required firmware.
      4In some cases the speed of the Mellanox interface can be shown as less then 100Gbps.N/A
      5Verify that the network can cause error message if the public network is running DHCP server.If the range (selected for the cloud above) is not overlapping with the DHCP pool, ignore this message. If an overlap exists, please fix it.
      6Launching more than three VMs at once might cause VM failures.Delete all extra configured VMs.
      7Migration of SR-IOV-based VMs is not working with nova migrate command.N/A
      8SR-IOV VMs will lose connectivity if the first port of the Mellanox NIC is down.Connectivity will be restored automatically when first port comes up, even if second port is the primary port.
      9In some cases Mellanox interfaces do not come up automatically when starting from bootstrap.

      Users can log in to a specific node and use command the ifup <INTERFACE_NAME> command.

      Use the ibdev2netdev command  to check for the correct interface name.