HowTo Configure Accelio's raio FIO Plugin

Version 10

    This post shows how to install and run Accelio's FIO plugin that supports the raio protocol (both client and server sides).

     

    Note: as PCIe over Fabrics spec was published, nbdX becomes obsolete. For more information about NVMe over Fabrics configuration, refer to HowTo Configure NVMe over Fabrics.

     

     

    References

     

    Prerequisites

    1. Please make sure to read the Accelio release notes and other documentation at Documentation | Accelio before you perform the steps described in this post.

    2. Install the latest MLNX_OFED package (can be downloaded from the OFED product page).

    3. Install extra development packages as follows:

     

    On Ubuntu:

    #apt-get install numactl libnuma-dev libaio-dev libevent-dev

     

    On RedHat

    #yum install numactl numactl-devel libaio-devel libevent-devel

     

    Compile FIO (Client side)

     

    1. Download fio from http://freecode.com/urls/3aa21b8c106cab742bf1f20d60629e3f  (fio – Freecode)

     

    2. Compile fio.

    # cd fio-2.1.10
    # ./configure
    ...
    # make
    ...
    # make install
    ...

     

    Procedure

    1. Download Accelio from the github.

    # git clone https://github.com/accelio/accelio.git
    ...
    # cd accelio

     

    2. Checkout the "fio_plugin-v1.1" tag of Accelio.

    # git checkout -b fio_plugin fio_plugin-v1.1

     

    3. Client side - Build and install the library.

    # make distclean

    # ./autogen.sh

    # export FIO_ROOT=<path to fio sources compiled in previous steps>

    # ./configure --enable-stat-counters=no --enable-extra-checks=no --enable-fio-build

    # make

    # make install

     

    4. Server side - Build and install the library.

    # make distclean

    # ./autogen.sh

    # ./configure --enable-stat-counters=no --enable-extra-checks=no

    # make

    # make install

     

    5. Add Accelio to the library's cache.

    # sh -c "echo '/opt/xio/lib' >> /etc/ld.so.conf.d/libxio.conf"

    # ldconfig

     

    6. Verity that RDMA is running by following the steps in the post: HowTo Enable, Verify and Troubleshoot RDMA

     

    A few important prerequisites for the subnet:

    • make sure opensm is running (only for ib link layer)
    • make sure that the client and server are on the same subnet

     

    Examples:

     

    Server Configuration

    1. Start the server application (raio_server) over IP address 11.222.82.14 and port 5555 for ib0 interface.

    # service irqbalance stop

    # set_irq_affinity_bynode.sh 0 ib0 # we are assuming that ib0 is close to NUMA 0 and the cpumask for ib0 is ff

    # raio_server -a 11.222.82.14 -p 5555 -c ff -t rdma -f 0 # you can add "-e 1" flag to run the server in polling mode instead of in interrupt mode. you can add "-n $(nproc)" to open thread per cpu (default is 6 threads)

     

    Client Configuration

    FIO example that runs against raio_server that we previously started (vs. null device as backing store).

    # cd accelio/examples/raio/usr/fio
    # taskset -c 1 $FIO_ROOT/fio --rw=randread --bs=4k --numjobs=1 --iodepth=64 --runtime=60 --time_based --loops=1 --ioengine=./.libs/libraio_fio.so --direct=1 --invalidate=1 --fsync_on_close=1 --randrepeat=1 --norandommap --group_reporting --exitall --name job1 --filename=/11.222.82.14/5555/dev/null --thread