HowTo Configure LIO enabled with ISER for Ubuntu 14.04 Inbox driver

Version 5

    This post is showing step by step, how to configure LIO target and initiator in iser transport mode using the Inbox driver.

     

    References

    Setup:

    • Make sure you have two servers equipped with Mellanox ConnectX-3 adapter cards
    • Connect the two servers via an Ethernet switch, you can use access port (VLAN 1 as default)
    • Install Ubuntu 14.04 Operating system on both servers

    Perquisites:

    Follow HowTo Setup RDMA Connection using Inbox Driver (RHEL, Ubuntu) post to make sure RoCE and RDMA is running between the servers.

     

    Note: The support for LIO iser target was added to MLNX-OFED -2.4 for rhel7, rhel6.4 with fbk16, ub14.04 and sles12. No changes were added to targetcli so no difference between the lio configuration process.

    The difference is only in the rdma layer (need to make sure that RDMA is enabled between the servers).

     

    LIO Target Configuration:

     

    1. Run the targetcli command to enter the LIO CLI console:

    # targetcli
    targetcli shell version 2.1.fb34
    Copyright 2011-2013 by Datera, Inc and others.
    For help on commands, type 'help'.

    /> ls

    o- / ............................................................................. [...]

      o- backstores .................................................................. [...]

      | o- fileio ....................................................... [0 Storage Object]

      | o- iblock ....................................................... [0 Storage Object]

      | o- pscsi ........................................................ [0 Storage Object]

      | o- rd_dr ........................................................ [0 Storage Object]

      | o- rd_mcp ....................................................... [0 Storage Object]

      o- ib_srpt ............................................................... [0 Targets]

      o- iscsi ................................................................. [0 Targets]

      o- loopback .............................................................. [0 Targets]

      o- qla2xxx ............................................................... [0 Targets]

      o- tcm_fc ................................................................ [0 Targets]

       

      

     

    2.  Create Ram Disk named test1:

    />   backstores/rd_mcp create  name=test1  size=100M
    Generating a wwn serial.
    Created rd_mcp ramdisk test1 with size 100M.

    /> ls
    o- / ............................................................................ [...]
      o- backstores ................................................................. [...]
      | o- fileio ...................................................... [0 Storage Object]
      | o- iblock ...................................................... [0 Storage Object]
      | o- pscsi ....................................................... [0 Storage Object]
      | o- rd_dr ....................................................... [0 Storage Object]
      | o- rd_mcp ...................................................... [1 Storage Object]
      |   o- test1 .................................................. [ramdisk deactivated]
      o- ib_srpt .............................................................. [0 Targets]
      o- iscsi ................................................................ [0 Targets]
      o- loopback ............................................................. [0 Targets]
      o- qla2xxx .............................................................. [0 Targets]
      o- tcm_fc ............................................................... [0 Targets]

       

     

     

     

    3. Create iscsi target:

     

    /> iscsi/ create wwn=iqn.2003-01.org.setup.lun.test

    Created target iqn.2003-01.org.setup.lun.test.

    Selected TPG Tag 1.

    Successfully created TPG 1.

    /> ls

    o- / ........................................................................... [...]

      o- backstores ................................................................ [...]

      | o- fileio ..................................................... [0 Storage Object]

      | o- iblock ..................................................... [0 Storage Object]

      | o- pscsi ...................................................... [0 Storage Object]

      | o- rd_dr ...................................................... [0 Storage Object]

      | o- rd_mcp ..................................................... [1 Storage Object]

      |   o- test1 ................................................. [ramdisk deactivated]

      o- ib_srpt ............................................................. [0 Targets]

      o- iscsi ................................................................ [1 Target]

      | o- iqn.2003-01.org.setup.lun.test ........................................ [1 TPG]

      |   o- tpgt1 ............................................................. [enabled]

      |     o- acls ............................................................. [0 ACLs]

      |     o- luns ............................................................. [0 LUNs]

      |     o- portals ....................................................... [0 Portals]

      o- loopback ............................................................ [0 Targets]

      o- qla2xxx ............................................................. [0 Targets]

      o- tcm_fc .............................................................. [0 Targets]

       

     

     

    4. Create LUN.

    Make sure that the ramdisk is activated when the LUN is created.

    /> /iscsi/iqn.2003-01.org.setup.lun.test/tpgt1/luns create /backstores/rd_mcp/test1

    Selected LUN 0.

    Successfully created LUN 0.

    /> ls

    o- / .......................................................................... [...]

      o- backstores ............................................................... [...]

      | o- fileio .................................................... [0 Storage Object]

      | o- iblock .................................................... [0 Storage Object]

      | o- pscsi ..................................................... [0 Storage Object]

      | o- rd_dr ..................................................... [0 Storage Object]

      | o- rd_mcp .................................................... [1 Storage Object]

      |   o- test1 .................................................. [ramdisk activated]

      o- ib_srpt ............................................................ [0 Targets]

      o- iscsi ............................................................... [1 Target]

      | o- iqn.2003-01.org.setup.lun.test ....................................... [1 TPG]

      |   o- tpgt1 ............................................................ [enabled]

      |     o- acls ............................................................ [0 ACLs]

      |     o- luns ............................................................. [1 LUN]

      |     | o- lun0 .......................................... [rd_mcp/test1 (ramdisk)]

      |     o- portals ...................................................... [0 Portals]

      o- loopback ........................................................... [0 Targets]

      o- qla2xxx ............................................................ [0 Targets]

      o- tcm_fc ............................................................. [0 Targets]

       

     

     

     

    5. Create portal. Make sure to supply the IP address of the interface you wish to run iSER on top (RoCE enabled).

     

    /> iscsi/iqn.2003-01.org.setup.lun.test/tpgt1/portals create 11.11.1.11

    Using default IP port 3260

    Successfully created network portal 11.11.1.11:3260.

    /> ls

    o- / ......................................................................... [...]

      o- backstores .............................................................. [...]

      | o- fileio ................................................... [0 Storage Object]

      | o- iblock ................................................... [0 Storage Object]

      | o- pscsi .................................................... [0 Storage Object]

      | o- rd_dr .................................................... [0 Storage Object]

      | o- rd_mcp ................................................... [1 Storage Object]

      |   o- test1 ................................................. [ramdisk activated]

      o- ib_srpt ........................................................... [0 Targets]

      o- iscsi ............................................................. [1 Targets]

      | o- iqn.2003-01.org.setup.lun.test ...................................... [1 TPG]

      |   o- tpgt1 ........................................................... [enabled]

      |     o- acls ........................................................... [0 ACLs]

      |     o- luns ............................................................ [1 LUN]

      |     | o- lun0 ......................................... [rd_mcp/test1 (ramdisk)]

      |     o- portals ...................................................... [1 Portal]

      |       o- 11.11.1.11:3260 ................................... [OK, iser disabled]

      o- loopback .......................................................... [0 Targets]

      o- qla2xxx ........................................................... [0 Targets]

      o- tcm_fc ............................................................ [0 Targets]

       

     

     

     

     

    6. Enable iSER on this portal

     

    /> iscsi/iqn.2003-01.org.setup.lun.test/tpgt1/portals/11.11.1.11:3260 iser_enable

    iser operation has been enabled

    /> ls

    o- / ......................................................................... [...]

      o- backstores .............................................................. [...]

      | o- fileio ................................................... [0 Storage Object]

      | o- iblock ................................................... [0 Storage Object]

      | o- pscsi .................................................... [0 Storage Object]

      | o- rd_dr .................................................... [0 Storage Object]

      | o- rd_mcp ................................................... [1 Storage Object]

      |   o- test1 ................................................. [ramdisk activated]

      o- ib_srpt ........................................................... [0 Targets]

      o- iscsi ............................................................. [1 Targets]

      | o- iqn.2003-01.org.setup.lun.test ...................................... [1 TPG]

      |   o- tpgt1 ........................................................... [enabled]

      |     o- acls ........................................................... [0 ACLs]

      |     o- luns ............................................................ [1 LUN]

      |     | o- lun0 ......................................... [rd_mcp/test1 (ramdisk)]

      |     o- portals ...................................................... [1 Portal]

      |       o- 11.11.1.11:3260 .................................... [OK, iser enabled]

      o- loopback .......................................................... [0 Targets]

      o- qla2xxx ........................................................... [0 Targets]

      o- tcm_fc ............................................................ [0 Targets]

       

     

     

    7. Allow access from any server (change ACL)

     

    /> iscsi/iqn.2003-01.org.setup.lun.test/tpgt1/ set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1

    Parameter demo_mode_write_protect is now '0'.

    Parameter authentication is now '0'.

    Parameter generate_node_acls is now '1'.

    Parameter cache_dynamic_acls is now '1'.

    /> ls

    o- / ......................................................................... [...]

      o- backstores .............................................................. [...]

      | o- fileio ................................................... [0 Storage Object]

      | o- iblock ................................................... [0 Storage Object]

      | o- pscsi .................................................... [0 Storage Object]

      | o- rd_dr .................................................... [0 Storage Object]

      | o- rd_mcp ................................................... [1 Storage Object]

      |   o- test1 ................................................. [ramdisk activated]

      o- ib_srpt ........................................................... [0 Targets]

      o- iscsi ............................................................. [1 Targets]

      | o- iqn.2003-01.org.setup.lun.test ...................................... [1 TPG]

      |   o- tpgt1 ................................................. [gen-acls, no-auth]

      |     o- acls .......................................................... [ACLs: 0]

      |     o- luns .......................................................... [LUNs: 1]

    |     | o- lun0 ......................................... [rd_mcp/test1 (ramdisk)]

      |     o- portals ...................................................... [1 Portal]

      |       o- 11.11.1.11:3260 .................................... [OK, iser enabled]

      o- loopback .......................................................... [0 Targets]

      o- qla2xxx ........................................................... [0 Targets]

      o- tcm_fc ............................................................ [0 Targets]

     

     

     

    8. Save configuration and exit:

    /> saveconfig

    Last 10 configs saved in /etc/target/backup.

    Configuration saved to /etc/target/saveconfig.json

    /> exit

    Global pref auto_save_on_exit=true

    Last 10 configs saved in /etc/target/backup.

    Configuration saved to /etc/target/saveconfig.json

       

    Initiator Configuration

    1. Make sure that ib_iser module is enabled:

    # lsmod | grep iser

    ib_iser                47000  2

    libiscsi               57278  1 ib_iser

    scsi_transport_iscsi   99754  3 ib_iser,libiscsi

    rdma_cm                43464  3 ib_iser,rdma_krping,rdma_ucm

    ib_core                87335  13 rdma_cm, ib_cm,ib_sa, iw_cm,mlx4_ib, ib_mad, ib_ucm, ib_iser, ib_umad, rdma_krping, ib_uverbs, rdma_ucm,ib_ipoib

       

     

    To enable it (in case it is not enabled), run:

    #modprobe ib_iser

       

    2. Run iscsiadm to discover the target and change the transport mode to iser

    # iscsiadm -m discovery -t st -p 11.11.1.11:3260

    11.11.1.11:3260,1 iqn.2003-01.org.setup.lun.test

    11.11.1.11:3260,1 iqn.2003-01.org.setup.lun.test

    # iscsiadm -m node -T iqn.2003-01.org.setup.lun.test -o update -n iface.transport_name -v iser

       

     

    3. Perform Login and try to read/write from the target (e.g. use dd command)

     

    # iscsiadm -m node -l

    Logging in to [iface: default, target: iqn.2003-01.org.setup.lun.test, portal: 11.11.1.11,3260] (multiple)

    Login to [iface: default, target: iqn.2003-01.org.setup.lun.test, portal: 11.11.1.11,3260] successful.

    # lsscsi

    [0:0:0:0]    disk    ATA      WDC WD1003FBYX-0 01.0  /dev/sda

    [7:0:0:0]    disk    LIO-ORG  test1            4.0   /dev/sdb

    [root@reg-r-vrt-002 ~]# dd if=/dev/sdb of=/dev/null bs=64k iflag=direct

    1600+0 records in

    1600+0 records out

    104857600 bytes (105 MB) copied, 0.0782739 s, 1.3 GB/s

       

     

    4. Run fio command (or similar) to test the link bandwidth

    # fio --rw=read --bs=128k --numjobs=1 --iodepth=4 --size="200m" --loops=1 --ioengine=libaio --direct=1 --invalidate=1 --fsync_on_close=1 --randrepeat=1 --norandommap --exitall --name "test1" --filename="/dev/sdb"  --output=/tmp/fio_verify.sh-22334.log --time_based --runtime=305

    Jobs: 1 (f=1): [R] [1.3% done] [4356MB/0KB/0KB /s] [34.6K/0/0 iops] [eta 05m:02s]