HowTo Install Accelio for nbdX

Version 25

    This post shows how to install Accelio for nbdX client and server.

     

    Note: as PCIe over Fabrics spec was published, nbdX becomes obsolete. For more information about NVMe over Fabrics configuration, refer to HowTo Configure NVMe over Fabrics.

     

    References

     

    Prerequisites

    1. Install the latest MLNX_OFED package (can be downloaded from Mellanox website)

    2. Install kernel 3.13.1 or higher on the client side (3.19 is recommended)

    3. Install extra development packages:

     

    Prior to start, please review the Accelio release notes and other documentation in here.

     

    On Ubuntu:

    #apt-get install numactl libnuma-dev libaio-dev libevent-dev

    ...

    On RedHat

    #yum install numactl numactl-devel libaio-devel libevent-devel libaio

    ...

     

    Compile FIO

     

    (build on client side, only if no fio tester is installed)

     

    1. Download fio from http://freecode.com/urls/3aa21b8c106cab742bf1f20d60629e3f  (fio – Freecode)

    # wget http://freecode.com/urls/3aa21b8c106cab742bf1f20d60629e3f

    ...

     

    2. Compile fio:

    # cd fio-2.1.10
    # ./configure
    ...
    # make
    ...
    # make install
    ...

     

    Installation

    1. Download Accelio from github

    # git clone https://github.com/accelio/accelio.git
    ...
    # cd accelio

     

    2. Checkout the "nbdx-v1.1" tag of Accelio.

    # git checkout -b nbdx_branch nbdx-v1.1

    Tag nbdx-v1.1 set up to track the latest nbdx stable version.

    Switched to a new branch 'nbdx_branch'

     

    3. Client side - Build and install the library with the kernel modules

     

    In case you try to install the package

    # ./autogen.sh

    benchmarks/usr/xio_perftest/Makefile.am: installing `./depcomp'

    # ./configure --enable-stat-counters=no --enable-extra-checks=no --enable-kernel-module

    ...

    # make

    ...

    # make install

    ...

     

    4. Server side - Build and install the library

    # make distclean

    # ./autogen.sh

    benchmarks/usr/xio_perftest/Makefile.am: installing `./depcomp'

     

    # ./configure --enable-stat-counters=no --enable-extra-checks=no

    ...

    # make

    ...

    # make install

    ...

     

    5. Load Accelio kernel modules (client side only)

    # modprobe xio_rdma; modprobe xio_tcp

     

     

    6. Add Accelio to library cache

    # sh -c "echo '/opt/xio/lib' >> /etc/ld.so.conf.d/libxio.conf"

    # ldconfig

     

    Verification

     

    1. Create two IP interfaces for both server and client hosts.

    In this example:

    • Server's IP 11.1.1.1/24
    • Client's IP: 11.1.1.2/24

     

    2. Follow this link to verity that RDMA is running HowTo Enable, Verify and Troubleshoot RDMA

     

    For example use rping between two hosts (client-server).

     

    Server host:

    #rping -sv -p 1234 -a 11.1.1.1

    Client host: (eth5 11.1.1.2/24)

    #rping -cv -p 1234 -a 11.1.1.1

     

    2. Verify that Accelio is running properly:

     

    Server machine:

    # /usr/local/bin/xio_server -c 1 -p 2061 11.1.1.1

    =============================================

    Server Address         : 11.1.1.1

    Server Port            : 2061

    Transport              : rdma

    Header Length          : 0

    Data Length            : 0

    Vector Length          : 1

    CPU Affinity           : 1

    =============================================

    listen to rdma://11.1.1.1:2061

    **** [0xa76f50] on_new_session :11.1.1.2:40549

    session event: new connection. session:0xa76f50, connection:0xa771a0, reason: Success

    **** message [4000000] hello world request header - hello world request data

    **** message [8000000] hello world request header - hello world request data

    **** message [12000000] hello world request header - hello world request data

    **** message [16000000] hello world request header - hello world request data

    **** message [20000000] hello world request header - hello world request data

    **** message [24000000] hello world request header - hello world request data

    **** message [28000000] hello world request header - hello world request data

     

     

    Client machine:

    # /usr/local/bin/xio_client -c 1 -p 2061 11.1.1.1

    =============================================

    Server Address         : 11.1.1.1

    Server Port            : 2061

    Transport              : rdma

    Header Length          : 32

    Data Length            : 32

    Out Vector Length      : 1

    In Vector Length       : 0

    Connection Index       : 0

    CPU Affinity           : 1

    Finite run             : 0

    =============================================

    warning - failed to allocate 2097152 bytes with hugepages. (errno=12 Cannot allocate memory)

    check that hugepages are configured. falling back to 4K pages allocations...

    **** starting ...

    **** [0x2000960] session established

    session event: connection established. reason: Success

    transactions per second: 980522, bandwidth: TX 59.85 MB/s, RX: 0.00 MB/s, length: TX: 64 B, RX: 0 B

    **** [2015/11/16-17:24:26.538943] - message [4000000] (null) - (null)

    transactions per second: 982549, bandwidth: TX 59.97 MB/s, RX: 0.00 MB/s, length: TX: 64 B, RX: 0 B

    **** [2015/11/16-17:24:30.610074] - message [8000000] (null) - (null)

    transactions per second: 982498, bandwidth: TX 59.97 MB/s, RX: 0.00 MB/s, length: TX: 64 B, RX: 0 B

    **** [2015/11/16-17:24:34.681367] - message [12000000] (null) - (null)

    transactions per second: 982406, bandwidth: TX 59.96 MB/s, RX: 0.00 MB/s, length: TX: 64 B, RX: 0 B

    **** [2015/11/16-17:24:38.753038] - message [16000000] (null) - (null)

    transactions per second: 982586, bandwidth: TX 59.97 MB/s, RX: 0.00 MB/s, length: TX: 64 B, RX: 0 B

    **** [2015/11/16-17:24:42.823963] - message [20000000] (null) - (null)

    transactions per second: 982366, bandwidth: TX 59.96 MB/s, RX: 0.00 MB/s, length: TX: 64 B, RX: 0 B

    **** [2015/11/16-17:24:46.895802] - message [24000000] (null) - (null)

    transactions per second: 982412, bandwidth: TX 59.96 MB/s, RX: 0.00 MB/s, length: TX: 64 B, RX: 0 B

     

    Note: this is not a performance test, but just a connection test. This is why the bandwidth you may see is ~60MB/s.

     

    By this point, Accelio is running on both hosts.

     

    At this point you can continue and configure NBDX here: HowTo Configure nbdX.

     

    Troubleshooting

    1. In some cases, there is a need to re-load the xio modules.

    To unload the modules run:

    # modprobe -r xio_tcp; modprobe -r xio_rdma; modprobe -r xio_core

     

    To load the modules run:

    # modprobe xio_rdma; modprobe xio_tcp

     

    2. In case you try to re-install the applicaiton, don't forget to delete the object files

     

    run:

    # make distclean

    ...