HowTo Launch VM over OVS-DPDK Using Mellanox OpenStack (devstack)

Version 9

    This post shows how to launch a Virtual Machine (VM) over OVS-DPDK using Mellanox ConnectX-4 adapters.

    In this example MLNX_OFED 3.4 was used.

    In this example we are also pinning 11 out of 12 CPUs for DPDK (host+guest)

     

    References

     

    Prerequisites

    1. Install MLNX_OFED, and using the ofed_info command verify that the version is correct:

    # ofed_info -s

    Note: DPDK 16.07 works with MLNX_OFED 3.3 and later.

               DPDK 16.11 works with MLNX_OFED 3.4 and later.

     

     

    Configuration

     

    Configure the grub File and Mount hugepages

    1. Update the grub.conf file.

    Note: Updating grub files is different for each Linux OS distribution. Refer to OS documentation.

     

    Edit the grub.conf file in the GRUB_CMDLINE_LINUX line as follows:

    "default_hugepagesz=2MB hugepagesz=2MB hugepages=14336 isolcpus=1-11 nohz_full=1-11 rcu_nocbs=1-11"

     

    The following line defines the hugepages size and quantity. For most Intel processors, 2Mb and 1G sizes are supported.  It is recommended that you leave the OS at least 2G RAM free. And isolates 11 CPUs for DPDK

     

    2. Update the grub script.

     

    3. Reboot the server. The configuration starts after you reboot.

     

    Devstack Configuration

    1. Add following lines to your local.conf

    # Enable Neutron Plugin

    enable_plugin neutron https://github.com/openstack/neutron

     

    # Enable OVS DPDK

    enable_plugin networking-ovs-dpdk https://github.com/openstack/networking-ovs-dpdk

    OVS_INTERFACE_DRIVER=mlnx

     

     

     

    OVS_NUM_HUGEPAGES=14336

    OVS_DPDK_MEM_SEGMENTS=2000

    OVS_HUGEPAGE_MOUNT_PAGESIZE=2M

    OVS_PMD_CORE_MASK=6

     

    # Set NUMA port

    OVS_SOCKET_MEM=2048,0

     

    RTE_TARGET=x86_64-native-linuxapp-gcc

     

     

    OVS_DATAPATH_TYPE=netdev

    OVS_BRIDGE_MAPPINGS="default:br-ovs-dpdk"

    OVS_GIT_TAG=v2.6.1

    OVS_DPDK_GIT_TAG=v16.11

     

     

     

    [[post-config|$NOVA_CONF]]

    [DEFAULT]

    firewall_driver=nova.virt.firewall.NoopFirewallDriver

    scheduler_default_filters=RamFilter,ComputeFilter,AvailabilityZoneFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,PciPassthroughFilter,NUMATopologyFilter

    vcpu_pin_set=5,6,7,8,9,10,11

     

     

    [libvirt]

    cpu_mode=host-model

     

    2. Run devstack:

    # ./stack.sh

    3. Add dpdk port to OVS:

    # sudo ovs-vsctl add-port br-ovs-dpdk dpdk0 -- set Interface dpdk0 type=dpdk

     

    Flavor Configuration

    1. Update nova flavor with cpu and memory parameters.

    # src: http://docs.openstack.org/admin-guide/compute-cpu-topologies.html

    # flavor_name=dpdk_4g4cpu

    # openstack flavor create $flavor_name --id auto --ram 4096 --disk 10 --vcpus 4

    # nova flavor-key $flavor_name set hw:mem_page_size=large

    # nova flavor-key $flavor_name set hw:cpu_mode=host-model

     

    VM Configuration

    1. Launch a guest machine.

    # nova boot --flavor $flavor_name --image $(glance image-list|grep mell | awk '{print $2}') --nic port-id=$(neutron port-create --binding:vnic_type=normal private | grep ' id   '|awk '{print $4}') vm1

     

    Installing DPDK in guest

    1. Download and install DPDK

    # wget  http://dpdk.org/browse/dpdk/snapshot/dpdk-16.11.zip

    # unzip dpdk-16.11.zip; cd dpdk-16.11

    # make -j install T=x86_64-native-linuxapp-gcc DESTDIR=install

    # modprobe uio

    # insmod igb_uio.k

    dpdk-16.11/dpdk-devbind.py --bind=igb_uio eth0

    Performance Tuning

    sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:pmd-cpu-mask=6

    sudo ovs-vsctl --no-wait set Open_vSwitch . other_config:dpdk-init=true

    sudo ovs-vsctl set Interface dpdk0 options:n_rxq=2 other_config:pmd-rxq-affinity="0:1,1:2"

    sudo ovs-vsctl set Interface vhost-user1 options:n_rxq=2 other_config:pmd-rxq-affinity="0:1,1:2"

    sudo ovs-vsctl add-port  br-ovs-dpdk vhost-user1 -- set Interface vhost-user1 type=dpdkvhostuser ofport_request=2

    Disable hyperthreading in server BIOS

     

     

     

     

    Knowing issues

    1. Devstack clean.sh not cleans all the OVS processes.

    for i in $(pgrep ovs-vswitchd); do sudo kill $i ; done