HowTo Configure LIO enabled with ISER for RHEL7 Inbox Driver

Version 18

    This post is showing step by step, how to configure LIO target and initiator in iser transport mode using the Inbox driver.

     

    References

    Setup

    • Make sure you have two servers equipped with Mellanox ConnectX-3 adapter cards
    • Connect the two servers via an Ethernet switch, you can use access port (VLAN 1 as default)
    • Install RHEL7 Operating system on both servers

     

    Perquisites

    Follow HowTo Setup RDMA Connection using Inbox Driver (RHEL, Ubuntu) post to make sure RoCE and RDMA is running between the servers.

     

    Note: The support for LIO iser target was added to MLNX-OFED -2.4 for rhel7, rhel6.4 with fbk16, ub14.04 and sles12. No changes were added to targetcli so no difference between the lio configuration process.

    The difference is only in the rdma layer (need to make sure that RDMA is enabled between the servers).

     

    Note: MLNX_OFED 3.1 supports includes ib_isert backport to RHEL7.1/CentOS7.1

     

    Security considerations

    It is recommended to disable various security threads that might block the traffic, for example disable the firewalld process.

    # service firewalld stop

    # systemctl disable firewalld

    # service iptables stop

     

    LIO Target Configuration

    Select the server to configure the Target CLI on. LIO is inbox in RHEL7, there is no need for additional installation.

     

    1. Run the targetcli command to enter the LIO CLI console:

    # targetcli
    targetcli shell version 2.1.fb34
    Copyright 2011-2013 by Datera, Inc and others.
    For help on commands, type 'help'.

    /> ls
    o- / ...................................................................................... [...]
      o- backstores ........................................................................... [...]
      | o- block ............................................................... [Storage Objects: 0]
      | o- fileio .............................................................. [Storage Objects: 0]
      | o- pscsi ............................................................... [Storage Objects: 0]
      | o- ramdisk ............................................................. [Storage Objects: 0]
      o- iscsi ......................................................................... [Targets: 0]
      o- loopback ...................................................................... [Targets: 0]
      o- srpt .......................................................................... [Targets: 0]

                         

     

    2.  Create Ram Disk named test1:

    /> backstores/ramdisk/ create  name=test1 nullio=true size=100M

    Created ramdisk test1 with size 100M.

    /> ls

    o- / ...................................................................................... [...]

      o- backstores ........................................................................... [...]

      | o- block ............................................................... [Storage Objects: 0]

      | o- fileio .............................................................. [Storage Objects: 0]

      | o- pscsi ............................................................... [Storage Objects: 0]

      | o- ramdisk ............................................................. [Storage Objects: 1]

      |   o- test1 .................................................. [nullio (100.0MiB) deactivated]

      o- iscsi ......................................................................... [Targets: 0]

      o- loopback ...................................................................... [Targets: 0]

      o- srpt .......................................................................... [Targets: 0]

                    

     

    3. Create iscsi target:

    /> iscsi/ create wwn=iqn.2003-01.org.setup.lun.test

    Created target iqn.2003-01.org.setup.lun.test.

    Created TPG 1.

    /> ls

    o- / ..................................................................................... [...]

      o- backstores .......................................................................... [...]

      | o- block .............................................................. [Storage Objects: 0]

      | o- fileio ............................................................. [Storage Objects: 0]

      | o- pscsi .............................................................. [Storage Objects: 0]

      | o- ramdisk ............................................................ [Storage Objects: 1]

      |   o- test1 ................................................. [nullio (100.0MiB) deactivated]

      o- iscsi ........................................................................ [Targets: 1]

      | o- iqn.2003-01.org.setup.lun.test ................................................ [TPGs: 1]

      |   o- tpg1 ........................................................... [no-gen-acls, no-auth]

      |     o- acls ...................................................................... [ACLs: 0]

      |     o- luns ...................................................................... [LUNs: 0]

      |     o- portals ................................................................ [Portals: 0]

      o- loopback ..................................................................... [Targets: 0]

      o- srpt ......................................................................... [Targets: 0]

                   

     

    4. Create LUN.

    Make sure that the ramdisk is activated when the LUN is created.

    /> iscsi/iqn.2003-01.org.setup.lun.test/tpg1/luns create /backstores/ramdisk/test1

    Created LUN 0.

    /> ls

    o- / .................................................................................... [...]

      o- backstores ......................................................................... [...]

      | o- block ............................................................. [Storage Objects: 0]

      | o- fileio ............................................................ [Storage Objects: 0]

      | o- pscsi ............................................................. [Storage Objects: 0]

      | o- ramdisk ........................................................... [Storage Objects: 1]

      |   o- test1 .................................................. [nullio (100.0MiB) activated]

      o- iscsi ....................................................................... [Targets: 1]

      | o- iqn.2003-01.org.setup.lun.test ............................................... [TPGs: 1]

      |   o- tpg1 .......................................................... [no-gen-acls, no-auth]

      |     o- acls ..................................................................... [ACLs: 0]

      |     o- luns ..................................................................... [LUNs: 1]

      |     | o- lun0 ............................................................. [ramdisk/test1]

      |     o- portals ............................................................... [Portals: 0]

      o- loopback .................................................................... [Targets: 0]

      o- srpt ........................................................................ [Targets: 0]

                  

     

    5. Create portal. Make sure to supply the IP address of the interface you wish to run iSER on top (RoCE enabled).

    /> iscsi/iqn.2003-01.org.setup.lun.test/tpg1/portals create 12.12.12.1

    Using default IP port 3260

    Created network portal 12.12.12.1:3260.

    /> ls

    o- / ................................................................................... [...]

      o- backstores ........................................................................ [...]

      | o- block ............................................................ [Storage Objects: 0]

      | o- fileio ........................................................... [Storage Objects: 0]

      | o- pscsi ............................................................ [Storage Objects: 0]

      | o- ramdisk .......................................................... [Storage Objects: 1]

      |   o- test1 ................................................. [nullio (100.0MiB) activated]

      o- iscsi ...................................................................... [Targets: 1]

      | o- iqn.2003-01.org.setup.lun.test .............................................. [TPGs: 1]

      |   o- tpg1 ......................................................... [no-gen-acls, no-auth]

      |     o- acls .................................................................... [ACLs: 0]

      |     o- luns .................................................................... [LUNs: 1]

      |     | o- lun0 ............................................................ [ramdisk/test1]

      |     o- portals .............................................................. [Portals: 1]

      |       o- 12.12.12.1:3260 ............................................................ [OK]

      o- loopback ................................................................... [Targets: 0]

      o- srpt ....................................................................... [Targets: 0]

                 

     

    6. Enable iSER on this portal (make sure ib_isert and mlx4_ib/mlx5_ib modules are loaded)

    /> iscsi/iqn.2003-01.org.setup.lun.test/tpg1/portals/12.12.12.1:3260 enable_iser boolean=true

    iSER enable now: True

    /> ls

    o- / .................................................................................. [...]

      o- backstores ....................................................................... [...]

      | o- block ........................................................... [Storage Objects: 0]

      | o- fileio .......................................................... [Storage Objects: 0]

      | o- pscsi ........................................................... [Storage Objects: 0]

      | o- ramdisk ......................................................... [Storage Objects: 1]

      |   o- test1 ................................................ [nullio (100.0MiB) activated]

      o- iscsi ..................................................................... [Targets: 1]

      | o- iqn.2003-01.org.setup.lun.test ............................................. [TPGs: 1]

      |   o- tpg1 ........................................................ [no-gen-acls, no-auth]

      |     o- acls ................................................................... [ACLs: 0]

      |     o- luns ................................................................... [LUNs: 1]

      |     | o- lun0 ........................................................... [ramdisk/test1]

      |     o- portals ............................................................. [Portals: 1]

      |       o- 12.12.12.1:3260 ......................................................... [iser]

      o- loopback .................................................................. [Targets: 0]

      o- srpt ...................................................................... [Targets: 0]

                

     

    7. Allow access from any server (change ACL)

    /> iscsi/iqn.2003-01.org.setup.lun.test/tpg1/ set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1

    Parameter demo_mode_write_protect is now '0'.

    Parameter authentication is now '0'.

    Parameter generate_node_acls is now '1'.

    Parameter cache_dynamic_acls is now '1'.

    /> ls

    o- / ................................................................................. [...]

      o- backstores ...................................................................... [...]

      | o- block .......................................................... [Storage Objects: 0]

      | o- fileio ......................................................... [Storage Objects: 0]

      | o- pscsi .......................................................... [Storage Objects: 0]

      | o- ramdisk ........................................................ [Storage Objects: 1]

      |   o- test1 ............................................... [nullio (100.0MiB) activated]

      o- iscsi .................................................................... [Targets: 1]

      | o- iqn.2003-01.org.setup.lun.test ............................................ [TPGs: 1]

      |   o- tpg1 .......................................................... [gen-acls, no-auth]

      |     o- acls .................................................................. [ACLs: 0]

      |     o- luns .................................................................. [LUNs: 1]

      |     | o- lun0 .......................................................... [ramdisk/test1]

      |     o- portals ............................................................ [Portals: 1]

      |       o- 12.12.12.1:3260 ........................................................ [iser]

      o- loopback ................................................................. [Targets: 0]

      o- srpt ..................................................................... [Targets: 0]

             

     

     

    8. Save configuration and exit:

    /> saveconfig

    Last 10 configs saved in /etc/target/backup.

    Configuration saved to /etc/target/saveconfig.json

    /> exit

    Global pref auto_save_on_exit=true

    Last 10 configs saved in /etc/target/backup.

    Configuration saved to /etc/target/saveconfig.json

                

    Initiator Configuration

    1. Make sure that ib_iser module is enabled:

    # lsmod | grep iser

    ib_iser                47000  2

    libiscsi               57278  1 ib_iser

    scsi_transport_iscsi   99754  3 ib_iser,libiscsi

    rdma_cm                43464  3 ib_iser,rdma_krping,rdma_ucm

    ib_core                87335  13 rdma_cm, ib_cm,ib_sa, iw_cm,mlx4_ib, ib_mad, ib_ucm, ib_iser, ib_umad, rdma_krping, ib_uverbs, rdma_ucm,ib_ipoib

               

     

    To enable it (in case it is not enabled), run:

    #modprobe ib_iser

               

    2. Run iscsiadm to discover the target and change the transport mode to iser

    # iscsiadm -m discovery -t st -p 12.12.12.1:3260

    12.12.12.1:3260,1 iqn.2003-01.org.setup.lun.test

    12.12.12.1:3260,1 iqn.2003-01.org.setup.lun.test

    # iscsiadm -m node -T iqn.2003-01.org.setup.lun.test -o update -n iface.transport_name -v iser

           

     

    3. Perform Login and try to read/write from the target (e.g. use dd command)

     

    # iscsiadm -m node -l

    Logging in to [iface: default, target: iqn.2003-01.org.setup.lun.test, portal: 12.12.12.1,3260] (multiple)

    Login to [iface: default, target: iqn.2003-01.org.setup.lun.test, portal: 12.12.12.1,3260] successful.

    # lsscsi

    [0:0:0:0]    disk    ATA      WDC WD1003FBYX-0 01.0  /dev/sda

    [7:0:0:0]    disk    LIO-ORG  test1            4.0   /dev/sdb

    [root@reg-r-vrt-002 ~]# dd if=/dev/sdb of=/dev/null bs=64k iflag=direct

    1600+0 records in

    1600+0 records out

    104857600 bytes (105 MB) copied, 0.0782739 s, 1.3 GB/s

           

     

    4. Run fio command (or similar) to test the link bandwidth

    # fio --rw=read --bs=128k --numjobs=1 --iodepth=4 --size="200m" --loops=1 --ioengine=libaio --direct=1 --invalidate=1 --fsync_on_close=1 --randrepeat=1 --norandommap --exitall --name "test1" --filename="/dev/sdb"  --output=/tmp/fio_verify.sh-22334.log --time_based --runtime=305

    Jobs: 1 (f=1): [R] [1.3% done] [4356MB/0KB/0KB /s] [34.6K/0/0 iops] [eta 05m:02s]