HowTo Install Mirantis Fuel 6.1 OpenStack with Mellanox Adapters Support (Ethernet Network)

Version 13

    This post shows how to set up and configure Mirantis Fuel 6.1 (OpenStack Juno based on CentOS 6.5) to support Mellanox ConnectX-3/ConnectX-3 Pro adapters. This procedure enables SR-IOV mode for the VMs on the compute nodes, and in iSER transport mode for the storage nodes. This post is based on HowTo Install Mirantis Fuel 6.0 OpenStack with Mellanox Adapters Support (Ethernet Network) with changes.

     

    For Infiniband 56Gb/s How-to manual, please, refer to the HowTo Install Mirantis Fuel 6.1 OpenStack with Mellanox Adapters Support (Infiniband Network).

     

    Related references

     

     

    Before reading this post, make sure you are familiar with Mirantis Fuel 6.1 installation procedures. It is also recommended to watch HowTo Install Mirantis Fuel 5.1 OpenStack with Mellanox Adapters video which is very similar.

     

     

    Setup Diagram

    setup_diagram_eth.png

     

     

    Note: Besides the Fuel Master node, all nodes should be connected to all five networks.

    Note: Server’s IPMI and the switches management interfaces wiring and configuration are out of scope. You need to ensure that there is management access (SSH) to Mellanox InfiniBand switch SX1710 to perform the configuration.

     

    Setup BOM (Example)

    ComponentQuantityDescription
    Fuel Master server
    1

    DELL PowerEdge R620

    • CPU: 2 x E5-2650 @ 2.00GHz
    • MEM: 128 GB
    • HD: 2 x 900GB SAS 10k in RAID-1
    Cloud Controllers and Compute servers:
    • 3 x Controllers
    • 3 x Computes
    6DELL PowerEdge R620
    • CPU: 2 x E5-2650 @ 2.00GHz
    • MEM: 128 GB
    • HD: 2 x 900GB SAS 10k in RAID-1
    • NIC: Mellanox ConnectX-3Pro VPI (MCX353-FCCT)
    Cloud Storage server1Supermicro X9DR3-F
    • CPU: 2 x E5-2650 @ 2.00GHz
    • MEM: 128 GB
    • HD: 24 x 6Gb/s SATA Intel SSD DC S3500 Series 480GB (SSDSC2BB480G4)
    • RAID Ctrl: LSI Logic MegaRAID SAS 2208 with battery
    • NIC: Mellanox ConnectX-3Pro VPI (MCX353-FCCT)
    Admin  (PXE) and Public switch11Gb switch with VLANs configured to support both networks
    Ethernet Switch1Mellanox SX1700 SDN 36 port switch configured in Ethernet mode.
    Cables

    16 x 1Gb CAT-6e for Admin (PXE) and Public networks

    7 x 56GbE copper cables up to 2m (MC2207130-XXX)

     

     

    Note: You can use Mellanox ConnectX-3 Pro EN (MCX313A-BCCT) or Mellanox ConnectX-3 Pro VPI (MCX353-FCCT) adapter cards.

    Note: Please make sure that the Mellanox switch is set as Ethernet.

    Storage server RAID Setup

    • 2 SSD drives in bays 0-1 configured in RAID-1 (Mirror) is used for the OS.
    • 22 SSD drives in bays 3-24 configured in RAID-10 is used as a Cinder volume and will be configured on the RAID drive.
      storage_raid.png

     

    Network Physical Setup

    1. Connect all nodes to the Admin (PXE) 1GbE switch (preferably through the eth0 interface on board).
        It is recommended to write the MAC address of the Controller and Storage servers to make Cloud installation easier (see Controller Node section below in Nodes tab).

    Note: All cloud servers should be configured to run PXE boot over the Admin (PXE) network.

     

    2. Connect all nodes to the Public 1GbE switch (preferably through the eth1 interface on board).

    3. Connect port #1 (eth0) of ConnectX-3 Pro to SX1710 Ethernet switch (Private, Management, Storage networks).

    Note: The interface names (eth0, eth1, p2p1, etc.) may vary between servers from different vendors.

    Note: Port bonding is not supported when using SR-IOV over the ConnectX-3 adapter family.

     

     

    Rack Setup Example:

    Rack Setup Example Eth.png

     

    Fuel node:

    Fuel.png

     

    Compute and Controller Nodes:

    controller-compute.png

     

    Storage Node:

    Storage.png

    4.  Configure the required VLANs and enable flow control on the Ethernet switch ports.

    All related VLANs should be enabled on the 40/56GbE switch (Private, Management, Storage networks).

    On Mellanox switches, use the command flow below to enable VLANs (e.g. VLAN 1-100 on all ports).

    Note: Refer to the MLNX-OS User Manual to get familiar with switch software (located at support.mellanox.com).

    Note: Before starting use of the Mellanox switch, it is recommended to upgrade the switch to the latest MLNX-OS version.

    switch > enable

    switch # configure terminal

    switch (config) # vlan 1-100

    switch (config vlan 1-100) # exit

    switch (config) # interface ethernet 1/1 switchport mode hybrid

    switch (config) # interface ethernet 1/1 switchport hybrid allowed-vlan all

    switch (config) # interface ethernet 1/2 switchport mode hybrid

    switch (config) # interface ethernet 1/2 switchport hybrid allowed-vlan all

    ...

    switch (config) # interface ethernet 1/36 switchport mode hybrid

    switch (config) # interface ethernet 1/36 switchport hybrid allowed-vlan all

    Flow control is required when running iSER (RDMA over RoCE - Ethernet). On Mellanox switches, run the following command to enable flow control on the switches (on all ports in this example):

    switch (config) # interface ethernet 1/1-1/36 flowcontrol receive on force

    switch (config) # interface ethernet 1/1-1/36 flowcontrol send on force

    To save the configuration (permanently), run:

    switch (config) # configuration write

    Note: Flow control (global pause) is normally enabled by default on the servers. If it is disabled, run:

    # ethtool -A <interface-name> rx on tx on

    Networks Allocation (Example)

    The example in this post is based on the network allocation defined in this table:

    NetworkSubnet/MaskGatewayNotes
    Admin (PXE)10.20.0.0/24N/AThe network is used to provision and manage Cloud nodes by the Fuel Master. The network is enclosed within a 1Gb switch and has no routing outside. 10.20.0.0/24 is the default Fuel subnet and we use it with no changes.
    Management192.168.0.0/24N/AThis is the Cloud Management network. The network uses VLAN 2 in SX1710 over 40/56Gb interconnect. 192.168.0.0/24 is the default Fuel subnet and we use it with no changes.
    Storage192.168.1.0/24N/AThis network is used to provide storage services. The network uses VLAN 3 in SX1710 over 40/56Gb interconnect. 192.168.1.0/24 is the default Fuel subnet and we use it with no changes.
    Public and Neutron L310.7.208.0/2410.7.208.1

    Public network is used to connect Cloud nodes to an external network.

    Neutron L3 is used to provide Floating IP for tenant VMs.

    Both networks are represented by IP ranges within same subnet with routing to external networks.

     

    All Cloud nodes will have Public IP address. In additional you shall allocate 2 more Public IP addressees ·

    • One IP required for HA functionality ·
    • From Fuel 6.1 virtual router requires additional Public IP address.

    We do not use virtual router in our deployment but still need to reserve Public IP address for it.   So Public Network range is an amount of cloud nodes + 2. For our example with 7 Cloud nodes we need 9 IPs in Public network range.

    Note: Consider a larger range if you are planning to add more servers to the cloud later.

     

    In our build we will use 10.7.208.53 >> 10.7.208.76 IP range for both Public and Neutron L3.

    IP allocation will be as follows:

    • Fuel Master IP: 10.7.208.53
    • Public Range: 10.7.208.54 >> 10.7.208.62 (7 used for physical servers, 1 reserved for HA and 1 reserved for virtual router)
    • Neutron L3 Range: 10.144.254.63 >> 10.144.254.76 (used for Floating IP pool)

     

    Below scheme illustrates our setup IP allocation.

    allocation_schema.png

     

    Install the Fuel Master from ISO Image:

    Note: Avoid starting the other nodes except for the fuel master until Mellanox plugin is installed.

    1. Boot Fuel Master Server from the ISO image as a virtual DVD (click here for the image).
    fuel_boot_menu.png

     

    2. After the reboot, boot from the local disk. The Fuel menu window will start.

    3. Network setup:

     

         a. Configure eth0 - PXE (Admin) network interface.

      Ensure the default Gateway entry is empty for the interface – the network is enclosed within the switch and has no routing outside. Select Apply.
    fuel_network_menu_a.png

    b. Configure eth1 – Public network interface.

        The interface is routable to LAN/internet and will be used to access the server. Configure static IP address, netmask and default gateway on the public network interface. Select Apply.
    fuel_network_menu_b.png

     

    4. Set the PXE Setup.
        The PXE network is enclosed within the switch. Do not make any changes, proceed with defaults.

        Press the Check button to ensure no errors are found.
    fuel_pxe.png

     

    5. Set the Time Sync.

    • Check NTP availability (e.g. 0.asia.pool.ntp.org) via Time Sync tab on the left.
    • Configure NTP server entries suitable for your infrastructure.
    • Press Check to verify settings.

    fuel_ntp.png

    6. Proceed with the installation.

    • Navigate to Quit Setup and select Save and Quit.

    fuel_save.png

       Once the Fuel installation is done, you are provided with Fuel access details both for SSH and HTTP.
    7. Access Fuel Web UI by http://10.7.208.53:8000. Use "admin" for both login and password.
        For SSH access use username root and password r00tme
    fuel_access.png

    Install Mellanox Plugin

    Mellanox plugin configures support for Mellanox ConnectX-3 Pro network adapters, enabling high-performance SR-IOV compute traffic networking, iSER (iSCSI) block storage networking which reduces CPU overhead, boosts throughput, reduces latency, and enables network traffic to bypass the software switch layer.

     

    Follow the steps below to install the plugin. For the complete instructions, please refer to: HowTo Install Mellanox OpenStack Plugin for Mirantis Fuel 6.1)

    1. Download the Mellanox plugin rpm from here and store it on your Fuel Master server

    2. Login to the fuel master and run the command below from the directory with the plugin:
    install_plugin.png

     

    Note: The Mellanox plugin replaces the current bootstrap image, the original image is backed up in /opt/old_bootstrap_image/

     

    3. Verify that the plugin was successfully installed.

        It should be displayed when running the fuel plugins command.
    Check_plugin.png

     

    4. Reboot of all discovered nodes.
        Run the fuel nodes command to check if there are any discovered nodes already.
    list_nodes.png
       Use buildin command # reboot_bootstrap_nodes to reboot discovered nodes.

     

     

    Creating a new OpenStack Environment

    Open in WEB browser (for example: http://10.7.208.53:8000) and log into Fuel environment using admin/admin as the username and password.

     

    1. Open a new environment in the Fuel dashboard. A configuration wizard will start.
    env_new.png

    2. Configure the new environment wizard as follows:

        • Name and Release
          • Name: TEST
          • Release: Icehouse on CentOS 6.5 (2014.1.1-5.1)
        • Compute
          • KVM
        • Network
          • Neutron with VLAN segmentation
        • Storage Backend
          • Cinder: Default
          • Glance : Default
        • Additional Services
          • None
        • Finish
          • Click Create button

    3. Click on the new TEST environment created and proceed with environment configuration.
    env_done.png

    Configuring the OpenStack Environment

     

    Settings Tab

    Mellanox Neutron and iSER storage component Public Network Assignment

    To work with SR-IOV mode, select Mellanox Openstack features, Neutron SR-IOV plugin.

    To work with iSER transport, select Mellanox Openstack features (if not selected before), iSER protocol for volumes (Cinder)

    Note: By default, the number of virtual NICs is 16. Please contact Mellanox support if you want to use more.

     

    iser.png

     

    Public Network Assignment

    Make sure Assign public network to all nodes is checked
    public_assign.png

    Save settings.
    save.png

    Nodes Tab
    nodes_tab.png

     

    Servers Discovery by Fuel

    This section assigns Cloud roles to servers. Servers should be discovered by Fuel, hence, make sure the servers are configured for PXE boot over Admin (PXE) network. When done, reboot the servers and wait for them to be discovered. Discovered nodes will be counted in top right corner of the Fuel dashboard.
    discovery.png
    Now you may add UNALLOCATED NODES to the setup.
    First you may add Controller, Storage, and then Compute nodes.

     

    Add Controller Nodes

    1. Click Add Node.

    2. Identify 3 controller node. Use the last 4 Hexadecimal digits of its MAC address of interface connected to Admin (PXE) network. Assign the node's role to be a Controller node.

    3. Click Apply Changes button
    add_ctrl.png

    Add Storage Node

    1. Click Add Node.

    2. Identify your storage node. Use the last 4 Hexadecimal digits of its MAC address of interface connected to Admin (PXE) network. In our example this is the only Supermicro server, so identification by vendor is easy. Select this node to be a Storage - Cinder node.

    3. Click Apply Changes button
    add_storage.png

    Add Compute Nodes

    1. Click Add Node.

    2. Select all the nodes that are left and assign them the Compute role.

    3. Click Apply Changes.

     

    Configure Interfaces

    In this step, each network must be mapped to a physical interface for each node. You can choose and configure multiple nodes in parallel.

    In case of HW differences between selected nodes (like the number of network ports), bulk configuration is not allowed. If you do a bulk configuration, the Configure Interfaces button will have an error icon (see below).

    conf_error.png

     

    The example below allows configuring 6 nodes in parallel. The 7th node (Supermicro storage node) will be configured separately.

    group_conf.png

     

    In this example, we set the Admin (PXE) network to eth0 and the Public network to eth1.

    The Storage, Private and Management networks should run on the ConnectX-3 adapters 40/56GbE port.

    conf_interfaces.png

     

    1. Click Back To Node List and perform network configuration for Storage Node.

    Note: Port bonding is not supported when using SR-IOV over ConnectX-3 Pro adapter family. There is no need to change the defaults for the Controller and Compute nodes unless the changes are required. For the Storage node it is recommended to allocate only high performing RAID as Cinder storage. The small disk shall be allocated to Base System.

    2. Select the Storage node.
    storage_node.png

    3. Press the Configure Disks button.
    conf_disks.png

    4. Click on the sda disk bar, set Cinder allowed space to 0 MB and make Base System occupy the entire drive – press USE ALL ALLOWED SPACE.
    conf_storage.png

    5. Click Apply.

     

    Networks Tab

    network_tab.png

    Public

    Note: In our example, Public network does not use VLAN. If you use VLAN for Public network you should check Use VLAN tagging and set proper VLAN ID.

    Public IP range: Consists of nodes IP range and Floating IP range. In our case, the full range is 10.7.208.54-10.7.208.76.

    IP range: Configure it to be part of your Public IP range, in this example, we select 10.7.208.54-10.7.208.62.

    Floating IP range: Configure it to be part of your Public IP range, in this example, we select 10.7.208.63-10.7.208.76.
    network_public.png

     

    Storage

    In this example, we select VLAN 3 for the storage network. The CIDR is left untouched.

    network_storage.png

     

    Management

    In this example, we select VLAN 2 for the management network. The CIDR is left untouched.

    network_management.png

     

    Neutron L2 Configuration

    In this example, we set the VLAN range to 4-100. It should be aligned with the switch VLAN configuration (above).
    The base MAC is left untouched.

    network_neutron.png

     

    Neutron L3 Configuration:

    Internal Network: Leave CIDR and Gateway with no changes.

    Name servers: Leave DNS servers with no changes.

    network_neutron_l3.png

    Save Configuration

    Click Save Settings at the bottom of page

     

    Verify Networks

    Click Verify Networks.
    You should see the following message: Verification succeeded. Your network is configured correctly. Otherwise, check the log file for troubleshooting.

    network_test_pass.png

     

    Note: If your public network runs a DHCP server, you can experience a verification failure as below. If the range selected for the cloud above is not overlapping with DHCP pool, you can ignore this message. If overlap exists, please fix it.

     

    network_test_fail.png


    Deployment

    Click the Deploy Changes button and view the installation progress at the nodes tab and view logs.

    deployment.png

     

    1. OS installation.

    deployment_os_install.png

     

    2. OS installation is finished.

    deployment_os_install_finished.png

     

    3. OPSTK installation on 1st controller.

    deployment_opstk_1st_ctrl.png

     

    4. OPSTK install on rest controllers.

    deployment_opstk_ctrls.png

     

    5. OPSTK install on Compute and Storage nodes.

    deployment_opstk_compute.png

     

    6. Installation completed.

    deployment_opstk_completed.png

    Health Test

    1. Click on the Health Test tab.
    health_check.png

    2. Check the Select All checkbox.

    3. Un-check Platform services functional tests (image with special packages is required).

    4. Un-check Launch instance, create snapshot, launch instance from snapshot in Functional tests group.
    5. Click Run Tests.

        All tests should pass. Otherwise, check the log file for troubleshooting.

     

    You can now safely use the cloud

    Click the dashboard link at the top of the page.

    use_cloud.png

     

    Usernames and Passwords:

    • Fuel server Dashboard user / password: admin / admin
    • Fuel server SSH user / password: root / r00tme
    • TestVM SSH user / password: cirros / cubswin:)
    • To get controller node CLI permissions run:  # source /root/openrc

     

    Prepare Linux VM Image for CloudX:

    In order to have network and RoCE support on the VM, MLNX_OFED (2.2-1 or later) should be installed on the VM environment.

    MLNX_OFED may be downloaded from http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers

    (In case of CentOS/RHEL OS, you can use virt-manager to open existing VM image and perform MLNX_OFED installation).

     

    Known Issues:

    Issue #

    Description

    Workaround

    Link to the Bug (in Launchpad)

    1

    The default number of supported virtual functions (VFs),16, is not sufficient.

    To have more vNICs available, contact Mellanox Support.

    3

    Snapshot creation of running instance fails

    To work this issue around, shut down the instance before taking a snapshot

    LP1398986

    4

    3rd party adapters based on the Mellanox chipset may not have SR-IOV enabled by default

    Apply to the device manufacturer for configuration instructions and for the required firmware.

     

    5

    Bonding is not supported for SR-IOV over the Mellanox ConnectX-3 network adapters family

     

     

    6

    Fuel is not starting deployment, it argues that Public IP range is not enough

    Increase Public IP range on 1.

    LP1487834