HowTo Install Mirantis Fuel 4.1 OpenStack with Mellanox Adapters Support

Version 70
    This post shows how to setup and configure Mirantis Fuel ver. 4.1 (OpenStack Havana based) to support Mellanox ConnectX-3 adapters to work in SR-IOV mode for the VMs on the compute nodes, and in iSER (iSCSI over RDMA) transport mode for the storage nodes.
    [Running Mirantis Fuel 5.1 .... check the configuration guide: HowTo Install Mirantis Fuel 5.1/5.1.1 OpenStack with Mellanox Adapters Support (Ethernet Network).]
    Related references:

     

    Before reading this post, Make sure you are familiar with Mirantis Fuel 4.1 installation procedures.

    Fuel3.png
    To understand the logical network design refer to Designing CloudX Solution using Mirantis Fuel OpenStack Software

    Note: Besides the Fuel node, all nodes should be connected to all five networks.

    Physical setup (bare metal connections and switch configurations):

    1. Connect all nodes to the Admin (PXE) 1GbE switch. Recommended via eth0 interface onboard.
    Its recommended to write the MAC address of Controller and Storage server to make Cloud instalation easier (see below Controller Node section in Nodes tab)
    2. Connect all nodes to the Public 1GbE switch. Recommended via eth1 interface onboard.
    3. Connect port #1 (eth2) of ConnectX-3 Pro adapter to Mellanox SX1036 Ethernet switch (management and private networks).
    4. Connect port #2 (eth3) of the ConnectX-3 Pro adapter to Mellanox SX1036 Ethernet switch (storage network).
    Note: The interface names (eth0, eth1 ...) may be different in various of servers.
    Here is an example taken from Fuel GUI:
    34.PNG.png
    5. Configure the required VLANs and enable flow-control on the ports on the Ethernet switch.
    All related VLANs should be enabled on the 40/56GbE switch (storage, management and private networks). In case using Mellanox switches, use the following commands to enable VLANs (e.g. VLAN 1-50 on all ports)
    Note: Make sure that you are familiar with MLNX-OS switch software, refer to the MLNX-OS UM in order to get started.
    Before start using the Mellanox switch, it is recommended to upgrade the switch to the latest MLNX-OS version.

    switch > enable

    switch # configure terminal

    switch (config) # vlan 1-20

    switch (config vlan 1-20) # exit

    switch (config) # interface ethernet 1/1 switchport mode hybrid

    switch (config) # interface ethernet 1/1 switchport hybrid allowed-vlan all

    switch (config) # interface ethernet 1/2 switchport mode hybrid

    switch (config) # interface ethernet 1/2 switchport hybrid allowed-vlan all

    ...

    switch (config) # interface ethernet 1/36 switchport mode hybrid

    switch (config) # interface ethernet 1/36 switchport hybrid allowed-vlan all

     

     

    Flow control is required when running iSER (RDMA over RoCE - Ethernet). When using Mellanox swiches run the following command to enable flowcontrol on the switches (e.g. on all ports):

    switch (config) # interface ethernet 1/1-1/36 flowcontrol receive on force

    switch (config) # interface ethernet 1/1-1/36 flowcontrol send on force

     

    To save the configuration (permanently), run:

    switch (config) # configuration write

     

    Note: Flow control (global pause) is normally enabled by default on the servers. In case it is disable, to enable it run:

     

    # ethtool -A <interface-name> rx on tx on

     

    6. In case you are running 56GbE, follow this example to set the link between the servers to the switch from 40GbE to 56GbE

    HowTo Configure 56GbE Link on Mellanox Adapters and Switches.

     

     

    Networks allocation (Example)

     

    The example in this post is based on the network allocation defined in this table:

    NetworkSubnet/MaskGatewwayNotes
    Admin (PXE)10.142.10.0/24N/A
    • The network is running in VLAN which is defined and encloswed within 1Gb switch. The network has no routing outside.
    • You need to choose IP address which will be assigned to Fuel Master. In our example it's 10.142.10.10
    Management192.168.0.0/24N/A
    • This is the default subnet by Fuel.
    • The network uses VLAN 2 in Mellanox SX1036 over 40/56Gb interconnect.
    Storage192.168.1.0/24N/A
    • This is the default subnet by Fuel.
    • The network is configured as untagged in Mellanox SX1036 over 40/56Gb interconnect.
    Public10.7.208.0/2410.7.208.1
    • The network is used to connect the cloud nodes and tenant VMs with the external network so routing is required.
    • You need to choose IP address which will be assigned to Fuel Master. In our example it's 10.7.208.10

     

     

    Install the Fuel Master via ISO image:

    1. (On the Fuel Master server) Boot from the ISO as a virtual CD (click here for the image), in order to install the Fuel master. For example, in SuperMicro load the ISO file via virtual storage and click on Plug-in button.
    2. Press the <TAB> key on the very first installation screen which says "Welcome to Fuel Installer" and update the kernel option from showmenu=no to showmenu=yes and click enter. It will now install Fuel and reboot the server.
    3. After the reboot, boot from the local disk. The Fuel menu window will start.
    4. Set PXE network attributes (e.g. on eth0)

    4.png
    5. In order to configure internet connection (via public network) on the Fuel Master, configure static IP address on the public network interface (e.g. on eth1), and configure the default netmask and default gateway Statically.
    5.png
    5. Set PXE ranges for the static and dynamic poll.
    6.png
    6. Check NTP availability (e.g. ntp.org) via Time Sync tab on the left.

    8.png

    6. Navigate to <Quit Setup> and click on save & quit button to proceed with the installation.
    7. Once the Fuel Installation is done, you can open your browser and go to http://<Fuel-Server-IP> (e.g. http://10.7.208.10)

    Nodes Discovery:

    1. Configure all setup nodes to boot from PXE on the interface connected to Admin (PXE) network (e.g. eth0) and reboot the nodes.
    2. The nodes will boot into bootstrap image from PXE running on Fuel master over Admin (PXE) network.
    3. The Bootstrap image will collect the information about the node and update the Fuel master. As result you'll see all your nodes in Fuel Dashboard.
    banner.PNG.png


    Creating a new OpenStack environment:

    1. Open a new environment in the Fuel dashboard. A Configuration wizard will start.
    32.PNG.png
    2. Configure the new environment wizard as follows:
    • Name and Release
      • Name: <any name>
      • OpenStack Release:  "Havana on Centos 6.4"
    • Deployment Mode
      • Multi-node
    • Compute
      • KVM

     

    • Network
      • Neutron with VLAN segmentation
    • Storage
      • Cinder - default
      • Glance - default
    • Additional Services
      • None
    • Click on "Create"

    33.PNG.png

     

    Configuring the Openstack environment:

    Settings Tab

    1. In order to work with SR-IOV mode, select "Install Mellanox plugin for Ethernet network" (otherwise Openvswitch plugin will be installed).
    2. In order to use high performance block storage, select "ISER protocol for volumes (Cinder)" checkbox in the storage section. Note: "Cinder LVM over iSCSI for volumes" should remain selected as well (default).44.PNG.png45.PNG.png
    3. Save the settings

     

    Configuring Cloud Networks

    Click on the Networks tab and set the following:

    1. Public:
      The same network will host IP addresses for physical Cloud servers as well as range for Cloud Floating IPs.
      In our example we have 10.7.208.20.0/24 subnet allocated for these purposes.IP addresses from
      10.7.208.20 till 10.7.208.100 will be assigned to servers and IP range from 10.7.208.101 till 10.7.208.200 will be used by Cloud Floating IP for the VMs.
      Net.Public.PNG.png
    2. Management:
      In this example, we select VLAN 2 for management network. The CIDR left untouched.
      Net.MNG.PNG.png
    3. Storage:
      In this example, we used untagged for the storage network. The CIDR left untouched.
      Net.Storage.PNG.png
    4. Neutron L2 Configuration:
      In this example, we set the VLAN range to be 3-50. It should be aligned with the switch VLAN configuration (above).
      The base MAC left untouched.
      vlans.png
    5. Neutron L3 Configuration:
      1. Floating IP range: configure it to be part of your Public network range, in this example, we select 10.7.208.101-10.7.208.200.
      2. Internal Network: Leave CIDR and Gateway with no changes
      3. Name servers: Leave DNS servers with no changes
        Net.L3.PNG.png

      6. Press Save Settings button at the bottom of page

     

     

    Assign Roles to Nodes:

    Now we'll add UNALLOCATED NODES to our setup.

    First we'll add Controller, then Stoarge and then Compute nodes.

     

    Click on the Nodes tab to configure Clous node roles as below.

    Controller Node:

    1. Click the "Add Node" button.
    2. Identify your controller node. Use last 4 hexa of its MAC address of interface connected to Admin (PXE) network.
      Select this node to be a controller node.
    3. Click on Apply Changes button.
      Add.Ctrl.2.PNG.png

     

    Storage Node:

    1. Click the "Add Node" button.
    2. Identify your controller node. Use last 4 hexa of its MAC address of interface connected to Admin (PXE) network.
      In our example this is an only Supermicro server so identification is easy.
      Select this node to be a Storage - Cinder LVM node.
    3. Click on Apply Changes button.
      Add.Stor.2.PNG.png

    Compute Nodes:

    1. Click the "Add Node" button.
    2. Select all left nodes and choose Compute Role to be assigne
    3. Click on Apply Changes button.

     

     

    Configure Interfaces

    In this step, we need to map each network to a physical interface for each node.

    configure_interfaces.PNG.png

    1. In this example, we set the Admin (PXE) network to eth0 and the Public network to eth1.
    2. The Cloud private and Management networks should run on the port #1of the connect-3 adapters 40/56GbE
    3. The storage network should be mapped on port #2.

     

     

    This is an example of the storage node:

    storage.PNG.png
    This is an example of the controller (or compute) node

    controller.PNG.png

    Configure Disks:

    There is no need to change the defaults here, unless you are sure changes are required.

    Refer to HowTo Configure iSER Block Storage for OpenStack Cloud with Mellanox ConnectX-3 Adapters for Storage server configuration.

     

    Verify Configuration and Deploy Changes

    1. Click on the Networks -> "Verify Network" button. You should see “Verification succeeded. Your network is configured correctly" message. Otherwise, you may check the log file for troubleshooting and check the known issues section below.
    2. Click on the "Deploy Changes" button, view the installation progress at the nodes tab and view logs.

     

    Health Test

    1. Click on the "Health Test" tab
    2. Check the "Select All" checkbox.
        - If using Mellanox neutron plugin, uncheck “Launch instance, create snapshot, launch instance from snapshot” (live migration and snapshot are not supported with running SRIOV VMs).
        - Uncheck “Check stack autoscaling” (image with special packages is required).
    3. Click on "Run Tests"
    (All should pass)
    42.PNG.png
    43.PNG.png

    You can now safely use the cloud (click on the link to dashboard at the top of the page).

    Usernames and Passwords:

    • Dashboard user: admin
    • Dashboard password: admin
    • MlnxVM user: root
    • MlnxVM password: password
    In order to get controller node CLI permissions run:
    #source /root/openrc

    Notes:

    1. When using Fuel 4.1, OFED 2.1 is installed via Fuel Master.
    2. Fuel uses OVS bridges for networking, which is a problem for RDMA based connections (such as RoCE), that’s why when using this ISO, in order to enable iSER, the user will need to have a dedicated port for storage network (will be probed VF in next releases).
    3. In this ISO multi node non-HA is supported. It was checked with Neutron-VLAN segmentation with iSER and MLNX/OVS Neutron plugins.

    Post Installation:

     

     

    The OpenStack is pre-installed with two images
    • MlnxVM (Provided with Mellanox drivers)
    • TestVM

     

    images.jpg

     

    Note: During Instance creation the MlnxVM must be used with Flavor medium or larger

     

    lunch_instance.jpg

    Known Issues:

     

    Issue #DescriptionWorkaroundBug in Launchpad
    1Cloud node is not discovered by Fuel after reboot

    dnsmasq service is not operating properly on Fuel
    Solution: restart the service on Fuel master: service dnsmasq restart

    2Network verification may fail sometimes with “untagged VLAN" warning.Try a few more times. This test should pass only Once in order to have the required indication.Already fixed in 5.0
    3

    Verify networks tool may be failed on some HW structures with the message:

    "[8830] Error running RPC method verify_networks: 3e5cd019-735d-41ea-a7a3-2380e123dcc4: MCollective call failed in agent 'net_probe', method 'start_frame_listeners', failed nodes:

    ID: 10 - Reason: Transport endpoint is not connected"

    In order to pass the Verify networks test, you should go to the failed nodes and run “killall -9 net_probe.py && service mcollective restart”.

     

    You may run below command on Fuel master to fix all cloud servers:

    for u in `arp -a -i eth0 | grep -v incomplete  | cut -d "(" -f 2 | cut
    -d")" -f 1` ; do echo -n "-I- Going to fix $u. Press CTRL+c to
    terminate or any key to proceed: " ; read zz ; ssh $u "killall -9
    net_probe.py && service mcollective restart"; done

    https://bugs.launchpad.net/fuel/+bug/1259935

    4

    Mellanox OEM adapter cards may be burned with SR-IOV disabled (by default).

    Contact Mellanox support to enable SR-IOV for OEM adapter cards.

     

    Note: Issues 1-2  should not affect the final installation in case of a proper connectivity and network configurations.