This post shows how to set up and configure Mirantis Fuel ver. 6.1 (OpenStack Juno based on CentOS 6.5) over InfiniBand 56Gb/s network with support for SR-IOV mode for the VMs on the compute nodes, and in iSER (iSCSI over RDMA) transport mode for the storage nodes.
For Ethernet 56Gb/s How-to manual, please, refer to the HowTo Install Mirantis Fuel 6.1 OpenStack with Mellanox Adapters Support (Ethernet Network).
- MLNX-OS User Manual - (located at support.mellanox.com )
- HowTo Install Mellanox OpenStack Plugin for Mirantis Fuel 6.1
- Planning Guide — Mirantis OpenStack v5.1 | Documentation
- Reference Architectures — Mirantis OpenStack v5.1 | Documentation
- HowTo Configure 56GbE Link on Mellanox Adapters and Switches
- HowTo upgrade MLNX-OS Software on Mellanox switches
- HowTo Configure iSER Block Storage for OpenStack Cloud with Mellanox ConnectX-3 Adapters
- Mellanox CloudX, Mirantis Fuel 5.1 Solution Guide
- HowTo Install Mirantis Fuel 6.1 OpenStack with Mellanox Adapters Support (Ethernet Network)
Before reading this post, make sure you are familiar with Mirantis Fuel 6.1 installation procedures. It is also recommended to watch HowTo Install Mirantis Fuel 5.1 OpenStack with Mellanox Adapters video.
Note: Server’s IPMI and the switches management interfaces wiring and configuration are out of scope.
You need to ensure that there is management access (SSH) to the Mellanox InfiniBand switch to perform the configuration.
|Fuel Master, Cloud Controllers and Compute servers:||7||DELL PowerEdge R620|
|Cloud Storage server||1||Supermicro X9DR3-F|
|Admin (PXE) and Public switch||1||1Gb switch with VLANs configured to support both networks|
|InfiniBand Switch||1||Mellanox SX1710 SDN 36 port switch configured in InfiniBand mode.|
16 x 1Gb CAT-6e for Admin (PXE) and Public networks
8 x 56GbE copper cables up to 2m (MC2207130-XXX)
Note: You can use Mellanox ConnectX-3 Pro EN (MCX313A-BCCT) or Mellanox ConnectX-3 Pro VPI (MCX353-FCCT) adapter cards.
Note: Please make sure that the Mellanox switch is set as InfiniBand.
Storage server RAID Setup
- 2 SSD drives in bays 0-1 configured in RAID-1 (Mirror): The OS will be installed on it.
- 22 SSD drives in bays 3-24 configured in RAID-10: The Cinder volume will be configured on the RAID drive.
Network Physical Setup
- Connect all nodes to the Admin (PXE) 1GbE switch (preferably through the eth0 interface on board).
It is recommended to write the MAC address of the Controller and Storage servers to make Cloud installation easier (see Controller Node section below in Nodes tab).
Note: All cloud servers should be configured to run PXE boot over the Admin (PXE) network.
2. Connect all nodes to the Public 1GbE switch (preferably through the eth1 interface on board).
3. Connect port #1 (ib0) of ConnectX-3 Pro to InfiniBand switch (Private, Management, Storage networks).
Note: The interface names (eth0, eth1, p2p1, etc.) may vary between servers from different vendors.
Note: Port bonding is not supported when using SR-IOV over the ConnectX-3 adapter family.
Note: Subnet manager (OpenSM) can run on any server that is connected to SwitchX. In this article OpenSM will run on Fuel node.
Rack Setup Example:
Fuel, Compute and Controller Nodes:
Networks Allocation (Example)
The example in this post is based on the network allocation defined in this table:
|Admin (PXE)||10.20.0.0/24||N/A||The network is used to provision and manage Cloud nodes by the Fuel Master. The network is enclosed within a 1Gb switch and has no routing outside. 10.20.0.0/24 is the default Fuel subnet and we use it with no changes.|
|Management||192.168.0.0/24||N/A||This is the Cloud Management network. The network uses VLAN 2 in SX1710 over 40/56Gb interconnect. 192.168.0.0/24 is the default Fuel subnet and we use it with no changes.|
|Storage||192.168.1.0/24||N/A||This network is used to provide storage services. The network uses VLAN 3 in SX1710 over 40/56Gb interconnect. 192.168.1.0/24 is the default Fuel subnet and we use it with no changes.|
|Public and Neutron L3||10.7.208.0/24||10.7.208.1|
Public network is used to connect Cloud nodes to an external network.
Neutron L3 is used to provide Floating IP for tenant VMs.
Both networks are represented by IP ranges within same subnet with routing to external networks.
All Cloud nodes will have Public IP address. In additional you shall allocate 2 more Public IP addressees ·
We do not use virtual router in our deployment but still need to reserve Public IP address for it. So Public Network range is an amount of cloud nodes + 2. For our example with 7 Cloud nodes we need 9 IPs in Public network range.
In our build we will use 10.7.208.53 >> 10.7.208.76 IP range for both Public and Neutron L3.
IP allocation will be as follows:
Below scheme illustrates our setup IP allocation.
Install the Fuel Master from ISO Image:
Note: Try to avoid start of other nodes except fuel master until Mellanox Plugin will be installed.
1. Boot Fuel Master Server from the ISO as a virtual DVD (click here for the image).
2. After the reboot, boot from the local disk. The Fuel menu window will start.
3. Network setup:
- Configure eth0 - PXE (Admin) network interface.
Ensure the default Gateway entry is empty for the interface – the network is enclosed within the switch and has no routing outside. Select Apply.
- Configure eth1 – Public network interface.
The interface is routable to LAN/internet and will be used to access the server. Configure static IP address, netmask and default gateway on the public network interface. Select Apply.
5. Set the Time Sync.
- Check NTP availability (e.g. 0.asia.pool.ntp.org) via Time Sync tab on the left.
- Configure NTP server entries suitable for your infrastructure.
- Press Check to verify settings.
Once the Fuel installation is done, you are provided with Fuel access details both for SSH and HTTP.
7. Access Fuel Web UI by http://10.7.208.53:8000. Use "admin" for both login and password.
For SSH access use username root and password r00tme
Install Mellanox Plugin
Mellanox plugin configures support for Mellanox ConnectX-3 Pro network adapters, enabling high-performance SR-IOV compute traffic networking, iSER (iSCSI) block storage networking which reduces CPU overhead, boosts throughput, reduces latency, and enables network traffic to bypass the software switch layer.
Follow below steps to install the plugin. For the complete instructions, please refer to: HowTo Install Mellanox OpenStack Plugin for Mirantis Fuel 6.1)
1. Download the Mellanox plugin rpm from here and store it on your Fuel Master server.
2. Log into the fuel master and run below command from the directory with the plugin:
Note: The Mellanox plugin replaces the current bootstrap image, the original image is backed up in /opt/old_bootstrap_image/
Since OpenSM runs on Fuel master, OFED should be installed on this server and configure PKEYs in partition.conf file. No extra configurations are required on the switch side.
Note: All the PKeys should be predefined in the partitions.conf file (/etc/opensm/partitions.conf). All PKeys are open on all servers ports (they get it from the SM configurations). The user can choose which VLAN he wants for which network, the mapping between VLAN to PKEY is done in the background.
1. Download MLNX_OFED 3.1 TGZ archive from MLNX_OFED Download Center to the Fuel node into /tmp directory
2. Install MLNX_OFED 3.1.
a. Install dependencies for MLNX_OFED.
# yum install kernel-devel python-devel redhat-rpm-config rpm-build gcc gtk2 atk cairo gcc-gfortran tcsh tcl bc tk
b. Go to /tmp folder and unpack the downloaded TGZ archive.
# tar -xzvf MLNX_OFED_LINUX-3.1-1.0.3-rhel6.5-x86_64.tgz
c. Go to the folder where MLNX_OFED was unpacked.
# cd MLNX_OFED_LINUX-3.1-1.0.3-rhel6.5-x86_64
d. Build customized MLNX_OFED binaries to support Fuel server kernel.
# ./mlnx_add_kernel_support.sh -m ./ --make-tgz
e. Unpack TGZ archive that you just created.
# tar -xzvf MLNX_OFED_LINUX-3.1-1.0.3-rhel6.5-x86_64-ext.tgz
f. Go to the folder where MLNX_OFED was unpacked.
# cd MLNX_OFED_LINUX-3.1-1.0.3-rhel6.5-x86_64-ext
g. Install MLNX_OFED.
# mlnxofedinstall –all
h. Reboot Fuel Master or use command:
# /etc/init.d/openibd restart
For more information, please refer to the official how-to.
InfiniBand OpenSM Configuration
We are going to create two files on Fuel node: /etc/opensm/partitions.conf and /etc/opensm/opensm.conf.
1. Create a new opensm.conf file.
# opensm -c /etc/opensm/opensm.conf
2. Enable virtualization by editing /etc/opensm/opensm.conf and changing the allow_both_pkeys value to TRUE.
3. Define the partition keys which are analog for Ethernet VLAN. Each VLAN will be mapped to one PK.
Add/Change the following with the command #vi /etc/opensm/partitions.conf file (Example)
management=0x7fff,ipoib, sl=0, defmember=full : ALL, ALL_SWITCHES=full,SELF=full;
vlan1=0x1, ipoib, sl=0, defmember=full : ALL;
vlan2=0x2, ipoib, sl=0, defmember=full : ALL;
vlan3=0x3, ipoib, sl=0, defmember=full : ALL;
vlan4=0x4, ipoib, sl=0, defmember=full : ALL;
vlan5=0x5, ipoib, sl=0, defmember=full : ALL;
vlan6=0x6, ipoib, sl=0, defmember=full : ALL;
vlan7=0x7, ipoib, sl=0, defmember=full : ALL;
vlan8=0x8, ipoib, sl=0, defmember=full : ALL;
vlan9=0x9, ipoib, sl=0, defmember=full : ALL;
vlan10=0xa, ipoib, sl=0, defmember=full : ALL;
. . .
vlan100=0x64, ipoib, sl=0, defmember=full : ALL;
In this example:
- VLAN2 assigned to PK 0x2 and will be used for Openstack Management network
VLAN3 assigned to PK 0x3 and will be used for Openstack Storage network
VLANs 4 through 100 are assigned to PKs 0x4 to 0x64 will be used for Tenant networks
Note: VLAN1 is defined, but not used for consistency with Ethernet setup installation.
Note: Maximum amount of VLANs is limited to 127.
4. Restart the OpenSM
# /etc/init.d/opensmd restart
Creating OpenStack Environment
Log into Fuel
1. Open a browser (for example: http://10.7.208.53:8000)
2. Log into Fuel using admin/admin for login/password.
Creating a new OpenStack Environment:
1. Open a new environment in the Fuel dashboard. A configuration wizard will start.
2. Configure the new environment wizard as follows:
- Name and Release
- Name: TEST
- Release: Icehouse on CentOS 6.5 (2014.1.1-5.1)
- Neutron with VLAN segmentation
- Storage Backend
- Cinder: Default
- Glance : Default
- Additional Services
- Click Create button
- When done, a new TEST environment will be created. Click on it and proceed with environment configuration.
Configuring the OpenStack Environment
Mellanox Neutron and iSER storage component Public Network Assignment
To work with SR-IOV mode select Mellanox Openstack features, Neutron SR-IOV plugin.
To work with iSER transport – select Mellanox Openstack features (if not selected before), iSER protocol for volumes (Cinder)
Note: By default, the number of virtual NICs is 16. Please contact Mellanox support if you want to use more.
Public Network Assignment
Servers Discovery by Fuel
This section will show how to assign Cloud roles to servers. To do so, the servers should be discovered by Fuel. Therefore, the servers must be configured for PXE boot over Admin (PXE) network. When done, reboot the servers and wait for them to be discovered. Discovered nodes will be counted in top right corner of the Fuel dashboard.
Now you may add UNALLOCATED NODES to the setup.
First you may add Controller, Storage, and then Compute nodes.
Verify Infiniband links for nodes
1. Access the Nodes tab in your environment.
2. Click on the cog wheel on the right of a node:
Add Controller Nodes
1. Click Add Node.
2. Identify 3 controller node. Use the last 4 Hexadecimal digits of its MAC address of interface connected to Admin (PXE) network. Assign the node's role to be a Controller node.
Add Storage Node
1. Click Add Node.
2. Identify your storage node. Use the last 4 Hexadecimal digits of its MAC address of interface connected to Admin (PXE) network. In our example this is the only Supermicro server, so identification by vendor is easy. Select this node to be a Storage - Cinder node.
Add Compute Nodes
1. Click Add Node.
2. Select all the nodes that are left and assign them the Compute role.
3. Click Apply Changes.
In this step, we will map each network to a physical interface for each node.You can choose and configure multiple nodes in parallel.
Fuel will not let you to proceed with bulk configuration if HW differences between selected nodes (like the number of network ports) are detected.
In this case the Configure Interfaces button will have an error icon (see below).
The example below allows configuring 6 nodes in parallel. The 7th node (Supermicro storage node) will be configured separately.
1. In this example, we set the Admin (PXE) network to eth0 and the Public network to eth1.
3. Click Back To Node List and perform network configuration for Storage Node
Note: Port bonding is not supported when using SR-IOV over ConnectX-3 Pro adapter family.
Configure Disks There is no need to change the defaults for the Controller and Compute nodes unless you are sure changes are required.
For the Storage node it is recommended to allocate only high performing RAID as Cinder storage.
The small disk shall be allocated to Base System.
6. Click on sda disk bar, set Cinder allowed space to 0 MB and make Base System occupy the entire drive – press USE ALL ALLOWED SPACE.
- Press Apply.
Note: In our example, Public network does not use VLAN. If you use VLAN for Public network, you should check Use VLAN tagging and set proper VLAN ID.
Public IP range: Consists of nodes IP range and Floating IP range. In our case, the full range is 10.7.208.54-10.7.208.76.
IP range: Configure it to be part of your Public IP range, in this example, we select 10.7.208.54-10.7.208.62.
Floating IP range: Configure it to be part of your Public IP range, in this example, we select 10.7.208.63-10.7.208.76.
In this example, we select VLAN 3 for the storage network. The CIDR is left untouched.
In this example, we select VLAN 2 for the management network. The CIDR left untouched.
Neutron L2 Configuration
In this example, we set the VLAN range to 4-100. It should be aligned with the switch VLAN configuration (above).
The base MAC is left untouched.
Neutron L3 Configuration:
Internal Network: Leave CIDR and Gateway with no changes.
Name servers: Leave DNS servers with no changes.
Click Save Settings at the bottom of page
Click Verify Networks.
You should see the following message: Verification succeeded. Your network is configured correctly. Otherwise, check the log file for troubleshooting.
Note: If your public network has running DHCP server, you can face verification failure as below.
If the range, selected for the cloud above is not overlapping with DHCP pool, you can ignore this message.
If overlap exists, please fix it
Click the Deploy Changes button and view the installation progress at the nodes tab and view logs.
Stage 1: OS installation
Stage 2: OS installation is finished
Stage 3: Install OPSTK on the 1st controller
Stage 4: Install OPSTK on the rest the controllers
Stage 5: Install OPSTK on the Compute and Storage nodes
Installation is completed
2. Check the Select All checkbox.
3. Un-check Platform services functional tests (image with special packages is required).
4. Un-check Launch instance, create snapshot, launch instance from snapshot in Functional tests group. (This is known issue that snapshot creation of running SRIOV instance fails. See Known Issues for more details)
5. Click Run Tests.
All tests should pass. Otherwise, check the log file for troubleshooting.
You can now safely use the cloud
Click the dashboard link at the top of the page.
Usernames and Passwords:
- Fuel server Dashboard user / password: admin / admin
- Fuel server SSH user / password: root / r00tme
- TestVM SSH user / password: cirros / cubswin:)
- To get controller node CLI permissions run: # source /root/openrc
Prepare Linux VM Image for CloudX:
In order to have network and RoCE support on the VM, MLNX_OFED (2.2-1 or later) should be installed on the VM environment.
MLNX_OFED may be downloaded from http://www.mellanox.com/page/products_dyn?product_family=26&mtag=linux_sw_drivers
(In case of CentOS/RHEL OS, you can use virt-manager to open existing VM image and perform MLNX_OFED installation).
|1||Configuring more than 16 VFs is not supported by default.||To have more vNICs available, please contact Mellanox Support.|
|2||Verify network can cause error message if public network has running DHCP server||If the range, selected for the cloud above is not overlapping with DHCP pool, please ignore this message. If overlap exists, please fix it.|
|3||Verify network before installation doesn’t support untagged VLAN verification and more than 60 PKEYs (shouldn’t affect installation).|
|4||Switch SM is not supported.||External server based SM node with configurations is required prior to the installation|
|5||Launching more than 5 VMs at once may cause eswitch allocation problems (eswitchd issue).||Delete all VMs and restart eswitchd.|
|6||In case of partitions.conf changes in the SM machines for "verify network" after failed verification, bootstrap nodes are not updated with the latest pkeys for another network verification.||Discovered nodes should be restarted for another network verification after PKEYs change (shouldn’t affect installation).|
|7||Migration of SR-IOV based VM is not working with Nova Migrate command.||Using snapshots.|
|8||Amount of VLANs is limited to 127|