HowTo Configure Docker in SR-IOV or Passthrough Mode with Mellanox Ethernet Adapters

Version 10

    Docker networking plugin for passthrough and SRIOV interfaces was released into docker git. This post provides configuration example for those modes for docker containers.

    This solution suits ConnectX-4 adapter cards and above.

     

    References

     

    Overview

     

    The docker-passthrough-plugin is a network plugin that allows direct (passthrough) access to the native Ethernet networking device from the Docker container(s).

    The plugin is supported for docker 1.9 and above. There is no known limitation on the OS or kernel.

    Note: RDMA/RoCE is not supported with this plugin, only native Ethernet is supported.

     

    Driver modes

    The plugin provides two modes of operations.

     

    sriov

    In this mode, the given netdev interface is used as PCIe physical function to define a network. All container instances will get one PCIe VF based network device when started. This mode uses PCIe SRIOV capability of the network devices.  SRIOV mode provides native access to the actual PCIe based networking device without any overheads of virtual devices. With this mode, every container can get dedicated NIC Tx and Rx queues to send and receive application data, without any contention to other containers. In addition, the plugin driver insures to enable or disable sriov while assigning VF based network device to container during container start.

     

    Note: When you create a network with sriov mode, the physical port (netdevice) can still be seen from the host (hypervisor) including all created VFs.

     

    passthrough

    In this mode, the given netdev interface is mapped to a container. This means that there is one network device per network, and therefore, every container gets one network. In some cases, there is a need to map bonded devices directly without additional layer and without consuming any extra MAC address.

     

    Note: When creating a network with passthrough mode, the physical port (netdevice) disappears from the host (hypervisor) and is given to the container.

     

    In some sense, both modes are similar to passthrough mode of KVM.

     

    With this plugin based interfaces, there is no limitation of IP address subnet for netdevice of container and netdevice of host. Any container can have any IP address, same or different subnet as that of host or other containers.

     

    Configuration

     

    Install Docker and Mellanox Plugin

     

    1. Install the latest Docker (Docker 1.9 or later).

    # yum install docker  -y

     

    2. Start Docker

    # service docker start

     

    3. Verify that the Docker version is above 1.9.

    # docker version

    Client:

    Version:         1.12.6

    API version:     1.24

    Package version: docker-1.12.6-28.git1398f24.el7.centos.x86_64

    Go version:      go1.7.4

    Git commit:      1398f24/1.12.6

    Built:           Fri May 26 17:28:18 2017

    OS/Arch:         linux/amd64

     

    Server:

    Version:         1.12.6

    API version:     1.24

    Package version: docker-1.12.6-28.git1398f24.el7.centos.x86_64

    Go version:      go1.7.4

    Git commit:      1398f24/1.12.6

    Built:           Fri May 26 17:28:18 2017

    OS/Arch:         linux/amd64

     

    4. Install (pull) the Mellanox passthrough-plugin

     

    # docker pull mellanox/passthrough-plugin

     

     

    5. Run the plugin.

    This command starts the container and emits console logs of the plugin in the terminal it was started in. It doesn't require the user/administrator to restart the docker engine.

     

    # docker run -v /run/docker/plugins:/run/docker/plugins --net=host --privileged mellanox/passthrough-plugin

     

    Open a new ssh (terminal) for that host to continue.

    6. Test Hello-World

    # docker run hello-world

    Unable to find image 'hello-world:latest' locally

    Trying to pull repository docker.io/library/hello-world ...

    latest: Pulling from docker.io/library/hello-world

     

    78445dd45222: Pull complete

    Digest: sha256:c5515758d4c5e1e838e9cd307f6c6a0d620b5e07e6f927b07d05f6d12a1ac8d7

     

    Hello from Docker!

    This message shows that your installation appears to be working correctly.

     

    To generate this message, Docker took the following steps:

    1. The Docker client contacted the Docker daemon.

    2. The Docker daemon pulled the "hello-world" image from the Docker Hub.

    3. The Docker daemon created a new container from that image which runs the

        executable that produces the output you are currently reading.

    4. The Docker daemon streamed that output to the Docker client, which sent it

        to your terminal.

     

    To try something more ambitious, you can run an Ubuntu container with:

    $ docker run -it ubuntu bash

     

    Share images, automate workflows, and more with a free Docker ID:

    https://cloud.docker.com/

     

    For more examples and ideas, visit:

    https://docs.docker.com/engine/userguide/

     

    7. Install Images, for example CentOS.

    # docker images

    REPOSITORY                              TAG                 IMAGE ID            CREATED             SIZE

    docker.io/mellanox/passthrough-plugin   latest              bd2a6b535d38        5 weeks ago         761.3 MB

    docker.io/hello-world                   latest              48b5124b2768        4 months ago        1.84 kB

     

    # docker run -it centos:latest bin/bash

    Unable to find image 'centos:latest' locally

    Trying to pull repository docker.io/library/centos ...

    latest: Pulling from docker.io/library/centos

     

    343b09361036: Pull complete

    Digest: sha256:bba1de7c9d900a898e3cadbae040dfe8a633c06bc104a0df76ae24483e03c077

     

     

    [root@e51e00a340d1 /]# exit

    exit

     

    # docker images

    REPOSITORY                              TAG                 IMAGE ID            CREATED             SIZE

    docker.io/centos                        latest              8140d0c64310        2 weeks ago         192.5 MB

    docker.io/mellanox/passthrough-plugin   latest              bd2a6b535d38        5 weeks ago         761.3 MB

    docker.io/hello-world                   latest              48b5124b2768        4 months ago        1.84 kB

     

    8. Run Docker commit on the image to keep the changes in the container.

    # docker images

    REPOSITORY                              TAG                 IMAGE ID            CREATED             SIZE

    docker.io/centos                        latest              8140d0c64310        2 weeks ago         192.5 MB

    docker.io/mellanox/passthrough-plugin   latest              bd2a6b535d38        5 weeks ago         761.3 MB

    docker.io/hello-world                   latest              48b5124b2768        4 months ago        1.84 kB

     

    # docker ps

    CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS               NAMES

    6d74780ed542        centos:latest                 "bin/bash"               4 minutes ago       Up 4 minutes                            sleepy_allen

    07a8731f6ffb        mellanox/passthrough-plugin   "docker-passthrough-p"   21 minutes ago      Up 21 minutes                           thirsty_albattani

     

    # docker commit 6d74780ed542                                                                                       <-- Take the container IS of the centos:latest

    sha256:15012f7cd8090a7d909923b829fed8c15189d58d24d45df40a0bb88d5c98caf4

     

    # docker images

    REPOSITORY                              TAG                 IMAGE ID            CREATED             SIZE

    <none>                                  <none>              15012f7cd809        4 seconds ago       192.5 MB       <-- This is a new Image (tagged) based on the CentOS latest image.

    docker.io/centos                        latest              8140d0c64310        2 weeks ago         192.5 MB

    docker.io/mellanox/passthrough-plugin   latest              bd2a6b535d38        5 weeks ago         761.3 MB

    docker.io/hello-world                   latest              48b5124b2768        4 months ago        1.84 kB

     

    9. Run the new tagged container, install net-tools and commit again to save this image.

     

    Run on one terminal:

    # docker images

    REPOSITORY                              TAG                 IMAGE ID            CREATED             SIZE

    <none>                                  <none>              15012f7cd809        3 minutes ago       192.5 MB

    docker.io/centos                        latest              8140d0c64310        2 weeks ago         192.5 MB

    docker.io/mellanox/passthrough-plugin   latest              bd2a6b535d38        5 weeks ago         761.3 MB

    docker.io/hello-world                   latest              48b5124b2768        4 months ago        1.84 kB

     

    Run on the second terminal:

    # docker run -it 15012f7cd809 bin/bash

    [root@d6eb09d97bfc /]# ifconfig

    bash: ifconfig: command not found

    [root@14c8f5b9507c /]# yum install net-tools

    ...

     

    Do not exit from the container yet. Switch back to the first terminal and commit, to keep the changes (latest yum install).

    Run on the first terminal:

    # docker ps

    CONTAINER ID        IMAGE                         COMMAND                  CREATED              STATUS              PORTS               NAMES

    14c8f5b9507c        15012f7cd809                  "bin/bash"               About a minute ago   Up About a minute                       thirsty_aryabhata

    07a8731f6ffb        mellanox/passthrough-plugin   "docker-passthrough-p"   26 minutes ago       Up 26 minutes                           thirsty_albattani

     

    # docker commit 14c8f5b9507c

    sha256:6307d86571e26cd4d0103ef4442d5511a3b6aa818974a4bfc0cafd7036d6657f

     

    Login again to that image, you should have the net-tools installed.

     

    Create a Container with SR-IOV mode

     

    1. Create a new network

     

    Below eth1 is PF based netdevice.

    • Set the mode is to sriov. The plugin driver will automatically assign a VF netdevice to container when it will start next.
    • The subnet could be any  different subnet than what eth1 has.
    • Give is a subnet name, for example mynet.

     

     

    # docker network create -d passthrough --subnet=194.168.1.0/24 -o netdevice=eth1 -o mode=sriov mynet
    7a69e3862c0773d88cc4fb6e4e70ff6d4e8fc720cc1bc0d823cd6bd0d35f163f

     

     

    2. Verify the created network.

    # docker network ls

    NETWORK ID          NAME                DRIVER              SCOPE

    19165e69962a        bridge              bridge              local              

    5da67399bc3b        host                host                local              

    7a69e3862c07        mynet               passthrough         local              

    06a3685f48bc        none                null                local       

     

    Note: the driver name is passthrough driver, the mode could be passthrough or sr-iov, it may be confusing as the mode is not listed in the network ls command.

     

    3. Run the container make use of passthrough-sriov interface.

    # docker images

    REPOSITORY                              TAG                 IMAGE ID            CREATED             SIZE

    <none>                                  <none>              6307d86571e2        11 minutes ago      286.2 MB

    docker.io/centos                        latest              8140d0c64310        2 weeks ago         192.5 MB

    docker.io/mellanox/passthrough-plugin   latest              bd2a6b535d38        5 weeks ago         761.3 MB

    docker.io/hello-world                   latest              48b5124b2768        4 months ago        1.84 kB

     

    # docker run --net=mynet -it 6307d86571e2  bin/bash        <-- use the IMAGE ID for the image we just saved.

     

    [root@af8d8e55271f /]# ifconfig

    eth0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500

            inet 194.168.1.2  netmask 255.255.255.0  broadcast 0.0.0.0     <-- Note that the IP address will be assgined by the docker from the network subnet (mynet) above.

            ether 96:75:05:cf:67:9c  txqueuelen 1000  (Ethernet)

            RX packets 0  bytes 0 (0.0 B)

            RX errors 0  dropped 0  overruns 0  frame 0

            TX packets 0  bytes 0 (0.0 B)

            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

     

    lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536

            inet 127.0.0.1  netmask 255.0.0.0

            inet6 ::1  prefixlen 128  scopeid 0x10<host>

            loop  txqueuelen 1  (Local Loopback)

            RX packets 0  bytes 0 (0.0 B)

            RX errors 0  dropped 0  overruns 0  frame 0

            TX packets 0  bytes 0 (0.0 B)

            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

     

    [root@af8d8e55271f /]#

     

    3. Verify, run ifconfig on the hypervisor host. you should be able to see eth1 (it is probed net device) and many new VF interfaces that were added.

     

    Create A Container with Passthrough mode

     

    This is very similar to SR-IOV mode, just change the mode to mode=passthrough in the creation of the network

     

    1. Create a new network

    Below eth2 is PF based netdevice.

    • Set the mode is to passthrough. The plugin driver will automatically assign a VF netdevice to container when it will start next.
    • The subnet could be any different subnet than what eth2 has.
    • Give is a subnet name, for example mynet2.

     

    # docker network create -d passthrough --subnet=194.168.2.0/24 -o netdevice=eth2  -o  mode=passthrough mynet2

    9799898d91b39740a1463f0ffea6206a901c71853b207a6190f599120e3c5925

     

    2. Verify the created network.

    # docker network ls

    NETWORK ID          NAME                DRIVER              SCOPE

    19165e69962a        bridge              bridge              local              

    5da67399bc3b        host                host                local              

    7a69e3862c07        mynet               passthrough         local              

    9799898d91b3        mynet2              passthrough         local              

    06a3685f48bc        none                null                local       

     

    Note: the driver name is passthrough driver, the mode could be passthrough or sr-iov, it may be confusing as the mode is not listed in the network ls command.

     

    3. Run the container to make use of passthrough mode.

    # docker images

    REPOSITORY                              TAG                 IMAGE ID            CREATED             SIZE

    <none>                                  <none>              6307d86571e2        27 minutes ago      286.2 MB

    docker.io/centos                        latest              8140d0c64310        2 weeks ago         192.5 MB

    docker.io/mellanox/passthrough-plugin   latest              bd2a6b535d38        5 weeks ago         761.3 MB

    docker.io/hello-world                   latest              48b5124b2768        4 months ago        1.84 kB

     

    # docker run --net=mynet2 -it 6307d86571e2  bin/bash                      <-- Use the network we just created (mynet2) and the image ID above.

     

    [root@20ec45fb8f0c /]# ifconfig

    eth0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500

            inet 194.168.2.2  netmask 255.255.255.0  broadcast 0.0.0.0        <-- Note that the IP is given automatically from the subnet configured to the network.

            ether 7c:fe:90:cb:76:03  txqueuelen 1000  (Ethernet)

            RX packets 0  bytes 0 (0.0 B)

            RX errors 0  dropped 0  overruns 0  frame 0

            TX packets 0  bytes 0 (0.0 B)

            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

     

     

    lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536

            inet 127.0.0.1  netmask 255.0.0.0

            inet6 ::1  prefixlen 128  scopeid 0x10<host>

            loop  txqueuelen 1  (Local Loopback)

            RX packets 0  bytes 0 (0.0 B)

            RX errors 0  dropped 0  overruns 0  frame 0

            TX packets 0  bytes 0 (0.0 B)

            TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

     

    4. Verify, run ifconfig on the hypervisor host. you should not be able to see eth2. it is hidden and assigned to the container now.