HowTo Enable T10-PI (T10-DIF) Data Integrity Protection in iSER with LIO Target

Version 27

    This post shows  how to bring-up basic setup with LIO target and initiator - withT10-PI DIF enabled with ISER support within Infiniband environment.

     

    References

     

    Setup and Prerequisites

     

    Note: LIO target and targetcli with rtslib installed on the "Target" server, it will be referenced as target in this post.

     

    Target Configuration

    1. Verify that on the target the configfs is mounted:

    # mount | grep configfs

    configfs on /sys/kernel/config type configfs (rw)

                           

    2. Enter targetcli mode and verify that there is no configuration on the target:

    # targetcli

    targetcli 2.1 (rtslib 2.2)

    Copyright (c) 2011-2013 by Datera, Inc.

    All rights reserved.

    /> ls

    o- / .............................................................................. [...]

      o- backstores ................................................................... [...]

      | o- fileio ........................................................ [0 Storage Object]

      | o- iblock ........................................................ [0 Storage Object]

      | o- pscsi ......................................................... [0 Storage Object]

      | o- rd_dr ......................................................... [0 Storage Object]

      | o- rd_mcp ........................................................ [0 Storage Object]

      o- ib_srpt ................................................................ [0 Targets]

      o- iscsi .................................................................. [0 Targets]

      o- loopback ............................................................... [0 Targets]

      o- tcm_fc ................................................................. [0 Targets]

                           

    3. Create target and verify that it was created:

    /> iscsi/ create

    Info: Created target iqn.2003-01.org.linux-iscsi.r-host82.x8664:sn.f88430f9299a.

    Info: Selected TPG Tag 1.

    Info: Successfully created TPG 1.

    /> ls

    o- / ................................................................................. [...]

      o- backstores ...................................................................... [...]

      | o- fileio ........................................................... [0 Storage Object]

      | o- iblock ........................................................... [0 Storage Object]

      | o- pscsi ............................................................ [0 Storage Object]

      | o- rd_dr ............................................................ [0 Storage Object]

      | o- rd_mcp ........................................................... [0 Storage Object]

      o- ib_srpt ................................................................... [0 Targets]

      o- iscsi ...................................................................... [1 Target]

      | o- iqn.2003-01.org.linux-iscsi.r-host82.x8664:sn.f88430f9299a .................. [1 TPG]

      |  o- tpgt1 ................................................................... [enabled]

      |    o- acls ................................................................... [0 ACLs]

      |    o- luns ................................................................... [0 LUNs]

      |    o- portals ............................................................. [0 Portals]

      o- loopback .................................................................. [0 Targets]

      o- tcm_fc .................................................................... [0 Targets]

                           

    4. The target should be able to be discovered (security issues). It can be done via either ACLs (as most people do), or with demo mode. The below output is an example that uses the demo mode (just to make things simple).

    /> /iscsi/iqn.2003-01.org.linux-iscsi.r-host82.x8664:sn.f88430f9299a/tpgt1/ set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1

    Parameter demo_mode_write_protect is now '0'.

    Parameter authentication is now '0'.

    Parameter generate_node_acls is now '1'.

    Parameter cache_dynamic_acls is now '1'.

                           

    5. Add t10_pi support

    /> /iscsi/iqn.2003-01.org.linux-iscsi.r-host82.x8664:sn.f88430f9299a/tpgt1/ set attribute t10_pi=1

    Parameter t10_pi is now '1'.

                           

    6. t10_pi is supported on target only with InitialR2T enabled and ImmediateData disabled. Configure the following parameters on the target:

    /> /iscsi/iqn.2003-01.org.linux-iscsi.r-host82.x8664:sn.f88430f9299a/tpgt1/ set parameter InitialR2T=Yes ImmediateData=No

    Parameter ImmediateData is now 'No'.

    Parameter InitialR2T is now 'Yes'.

                           

    7. Configure the portals that will be available for initiators to discover the target:

    /> /iscsi/iqn.2003-01.org.linux-iscsi.r-host82.x8664:sn.f88430f9299a/tpgt1/portals create 11.209.12.152 ip_port=3260

    Info: Successfully created network portal 11.209.12.152:3260.

                           

    8. The portal will support regular iSCSI traffic - non RDMA. enable iSER to support RDMA capabilities:

    /> /iscsi/iqn.2003-01.org.linux-iscsi.r-host82.x8664:sn.f88430f9299a/tpgt1/portals/11.209.12.152:3260 enable_iser boolean=true

    Info: iser operation has been enabled

                           

    Now after iSER enabled on the portal the target should look like:

    /> ls

    o- / ............................................................................. [...]

      o- backstores .................................................................. [...]

      | o- fileio ....................................................... [0 Storage Object]

      | o- iblock ....................................................... [0 Storage Object]

      | o- pscsi ........................................................ [0 Storage Object]

      | o- rd_dr ........................................................ [0 Storage Object]

      | o- rd_mcp ....................................................... [0 Storage Object]

      o- ib_srpt ............................................................... [0 Targets]

      o- iscsi .................................................................. [1 Target]

      | o- iqn.2003-01.org.linux-iscsi.r-host82.x8664:sn.f88430f9299a .............. [1 TPG]

      |  o- tpgt1 ................................................................ [enabled]

      |    o- acls ................................................................ [0 ACLs]

      |    o- luns ................................................................ [0 LUNs]

      |    o- portals ........................................................... [1 Portal]

      |      o- 11.209.12.152:3260 ................................................... [iser]

      o- loopback .............................................................. [0 Targets]

      o- tcm_fc ................................................................ [0 Targets]

          

     

    9. Create a PI capable device. It can be done via:

    RAM device(rd_mcp), file(fileio), block device(iblock) or scsi_debug(iblock).

    In this example, it is shown via fileio device as it is the most visual device:

    /> /backstores/fileio create file_or_dev=/tmp/difSupportedDev name=dev01 size=1G

    Info: Generating a wwn serial.

    Info: Not using buffered mode.

    Info: Created fileio dev01.

                           

     

    Note: When using block device, scsi_debug or RAM devices, no need to format the devices. Moreover, when using scsi_debug or block device, no need to specify the pi_prot_type.

    10. Enable and format the following parameters, in case of scsi_debug or real block device, skip this step:

    /> /backstores/fileio/dev01 set attribute pi_prot_type=1

    Parameter pi_prot_type is now '1'.

    /> /backstores/fileio/dev01 set attribute pi_prot_format=1

    Parameter pi_prot_format is now '0'.

                           

              * order is important, set protection type before formatting

    11. Add the device to the target just created:

    /> /iscsi/iqn.2003-01.org.linux-iscsi.r-host82.x8664:sn.f88430f9299a/tpgt1/luns create /backstores/fileio/dev01

    Info: Selected LUN 0.

    Info: Successfully created LUN 0.

                           

    12. Verify that the fileio device created a protection file for the protection information to be saved (exit targetcli mode to do that):

    # ll /tmp/

    total 16408

    -rw------- 1 root root        0 Oct 21 10:59 difSupportedDev

    -rw------- 1 root root 16777216 Oct 21 11:00 difSupportedDev.protection

                           

    13. See the content of the protection file:

    # hexdump -C /tmp/difSupportedDev.protection

    00000000  ff ff ff ff ff ff ff ff  ff ff ff ff ff ff ff ff  |▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒|

    *

    01000000

                           

     

    Because the device is formatted, there is no data - it is filled with ffff.

    The target is ready, run ls to see the final configuration:

    # targetcli

    targetcli 2.1 (rtslib 2.2)

    Copyright (c) 2011-2013 by Datera, Inc.

    All rights reserved.

    /> ls

    o- / ............................................................................. [...]

      o- backstores .................................................................. [...]

      | o- fileio ....................................................... [1 Storage Object]

      | | o- dev01 ........................................ [/tmp/difSupportedDev activated]

      | o- iblock ....................................................... [0 Storage Object]

      | o- pscsi ........................................................ [0 Storage Object]

      | o- rd_dr ........................................................ [0 Storage Object]

      | o- rd_mcp ....................................................... [0 Storage Object]

      o- ib_srpt ............................................................... [0 Targets]

      o- iscsi .................................................................. [1 Target]

      | o- iqn.2003-01.org.linux-iscsi.r-host82.x8664:sn.f88430f9299a .............. [1 TPG]

      |  o- tpgt1 ................................................................ [enabled]

      |    o- acls ................................................................ [0 ACLs]

      |    o- luns ................................................................. [1 LUN]

      |    | o- lun0 ................................. [fileio/dev01 (/tmp/difSupportedDev)]

      |    o- portals ........................................................... [1 Portal]

      |      o- 11.209.12.152:3260 ...................................... [OK, iser enabled]

      o- loopback .............................................................. [0 Targets]

      o- tcm_fc ................................................................ [0 Targets]

                           

     

     

    Initiator Configuration

     

    1. Load the iSER module with pi support, if it is currently loaded on the host, check if it pi_enabled:

    # cat /sys/module/ib_iser/parameters/pi_enable

    Y

                           

    2. If it is not loaded "N" - the default for iSER, unload it, and re-load it with pi_enable parameter as Y:

    After loading it with pi_enable, verify it is as expected:

    # cat /sys/module/ib_iser/parameters/pi_enable

    N

    # modprobe -r ib_iser

    # modprobe ib_iser pi_enable=1

    # cat /sys/module/ib_iser/parameters/pi_enable

    Y

                           

    * If you have troubles unloading ib_iser, please make sure all iSER connection are closed and multipath is flushed, and there are no refcounts on the ib_iser module.

    3. Connect to target:

    # iscsiadm -m discovery -t st -I iser -p 11.209.12.152 -l

    11.209.12.152:3260,1 iqn.2003-01.org.linux-iscsi.r-host82.x8664:sn.f88430f9299a

    Logging in to [iface: iser, target: iqn.2003-01.org.linux-iscsi.r-host82.x8664:sn.f88430f9299a, portal: 11.209.12.152,3260] (multiple)

    Login to [iface: iser, target: iqn.2003-01.org.linux-iscsi.r-host82.x8664:sn.f88430f9299a, portal: 11.209.12.152,3260] successful.

                           


    Verification

     

    1. Check the LUN on initiator:

    # iscsiadm -m session -P3

    iSCSI Transport Class version 2.0-870

    version 6.2.0-873.10.el6

    Target: iqn.2003-01.org.linux-iscsi.r-host82.x8664:sn.f88430f9299a (non-flash)

            Current Portal: 11.209.12.152:3260,1

            Persistent Portal: 11.209.12.152:3260,1

                    **********

                    Interface:

                    **********

                    Iface Name: iser

                    Iface Transport: iser

                    Iface Initiatorname: iqn.1988-12.com.oracle:39c87cf9ce8

                    Iface IPaddress: <empty>

                    Iface HWaddress: <empty>

                    Iface Netdev: <empty>

                    SID: 371

                    iSCSI Connection State: LOGGED IN

                    iSCSI Session State: LOGGED_IN

                    Internal iscsid Session State: NO CHANGE

                    *********

                    Timeouts:

                    *********

                    Recovery Timeout: 60

                    Target Reset Timeout: 30

                    LUN Reset Timeout: 30

                    Abort Timeout: 15

                    *****

                    CHAP:

                    *****

                    username: <empty>

                    password: ********

                    username_in: <empty>

                    password_in: ********

                    ************************

                    Negotiated iSCSI params:

                    ************************

                    HeaderDigest: None

                    DataDigest: None

                    MaxRecvDataSegmentLength: 128

                    MaxXmitDataSegmentLength: 8192

                    FirstBurstLength: 65536

                    MaxBurstLength: 262144

                    ImmediateData: No

                    InitialR2T: Yes

                    MaxOutstandingR2T: 1

                    ************************

                    Attached SCSI devices:

                    ************************

                    Host Number: 377        State: running

                    scsi377 Channel 00 Id 0 Lun: 0

                            Attached scsi disk sdb          State: running

                           

    2. Check the logs, that the device is DIF enabled:

    sd 379:0:0:0: Attached scsi generic sg3 type 0

    sd 379:0:0:0: [sdc] Enabling DIF Type 1 protection

    sd 379:0:0:0: [sdc] 2097152 512-byte logical blocks: (1.07 GB/1.00 GiB)

    sd 379:0:0:0: [sdc] Write Protect is off

    sd 379:0:0:0: [sdc] Mode Sense: 43 00 00 08

    sd 379:0:0:0: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA

    sdc: unknown partition table

    sd 379:0:0:0: [sdc] Enabling DIX T10-DIF-TYPE1-CRC protection

    sd 379:0:0:0: [sdc] DIF application tag size 2

    sd 379:0:0:0: [sdc] Attached SCSI disk

    device-mapper: table: 252:0: integrity not set: sdc and sdb profile mismatch

                           

    3. Query the devices attributes:

    # cat /sys/block/sdc/integrity/format

    T10-DIF-TYPE1-CRC

                          

    4. Query the device with sg query (for that sg3_utils should be installed):

    # sg_readcap /dev/sdc --long

    Read Capacity results:

      Protection: prot_en=1, p_type=0, p_i_exponent=0

      Thin provisioning: tpe=0, tprz=0

      Last logical block address=2097151 (0x1fffff), Number of logical blocks=2097152

      Logical block length=512 bytes

      Logical blocks per physical block exponent=0

      Lowest aligned logical block address=0

    Hence:

      Device size: 1073741824 bytes, 1024.0 MiB, 1.07 GB

                          

     

    5. Perform I/O to the device:

    # dd if=/dev/urandom of=/dev/sdc count=10 bs=128k oflag=direct

    10+0 records in

    10+0 records out

    1310720 bytes (1.3 MB) copied, 0.291316 s, 4.5 MB/s

                          

     

    6. Go back to the target and see that the protection file have changed and have some protection information for us:

    # hexdump -C /tmp/difSupportedDev.protection

    00000000  18 a1 00 00 00 00 00 00  64 62 00 00 00 00 00 01  |.▒......db......|

    00000010  0e 83 00 00 00 00 00 02  43 59 00 00 00 00 00 03  |........CY......|

    00000020  37 d1 00 00 00 00 00 04  8f 20 00 00 00 00 00 05  |7▒....... ......|

    00000030  3b 66 00 00 00 00 00 06  b9 0c 00 00 00 00 00 07  |;f......▒.......|

    00000040  73 59 00 00 00 00 00 08  af b1 00 00 00 00 00 09  |sY......▒▒......|

    00000050  ed c8 00 00 00 00 00 0a  82 63 00 00 00 00 00 0b  |▒▒.......c......|

    00000060  42 3b 00 00 00 00 00 0c  cc 77 00 00 00 00 00 0d  |B;......▒w......|

    00000070  e2 7f 00 00 00 00 00 0e  6a 78 00 00 00 00 00 0f  |▒.......jx......|

    00000080  75 a3 00 00 00 00 00 10  f7 a4 00 00 00 00 00 11  |u▒......▒......|

    00000090  b5 4d 00 00 00 00 00 12  e0 00 00 00 00 00 00 13  |▒M......▒.......|

    000000a0  38 47 00 00 00 00 00 14  1c 28 00 00 00 00 00 15  |8G.......(......|

    000000b0  5c 34 00 00 00 00 00 16  d0 df 00 00 00 00 00 17  |\4......▒▒......|

    000000c0  c0 74 00 00 00 00 00 18  d2 56 00 00 00 00 00 19  |▒t......▒V......|

    000000d0  eb d1 00 00 00 00 00 1a  91 d5 00 00 00 00 00 1b  |▒▒.......▒......|

    000000e0  5b 21 00 00 00 00 00 1c  fd 4d 00 00 00 00 00 1d  |[!......▒M......|

    000000f0  83 20 00 00 00 00 00 1e  39 3c 00 00 00 00 00 1f  |. ......9<......|

    00000100  52 82 00 00 00 00 00 20  fd 88 00 00 00 00 00 21  |R...... ▒......!|

    00000110  40 61 00 00 00 00 00 22  2f 94 00 00 00 00 00 23  |@a....."/......#|

    00000120  a2 c2 00 00 00 00 00 24  00 c4 00 00 00 00 00 25  |▒▒.....$.▒.....%|

    00000130  f3 da 00 00 00 00 00 26  26 23 00 00 00 00 00 27  |▒▒.....&&#.....'|

    00000140  ba 8f 00 00 00 00 00 28  11 f3 00 00 00 00 00 29  |▒......(.▒.....)|

    00000150  5c a9 00 00 00 00 00 2a  c1 b9 00 00 00 00 00 2b  |\▒.....*▒.....+|

    00000160  4c 9b 00 00 00 00 00 2c  01 a4 00 00 00 00 00 2d  |L......,.▒.....-|

    00000170  7e 68 00 00 00 00 00 2e  b8 a1 00 00 00 00 00 2f  |~h......▒▒...../|

    00000180  de b4 00 00 00 00 00 30  c4 33 00 00 00 00 00 31  |޴.....0▒3.....1|

    00000190  99 00 00 00 00 00 00 32  81 af 00 00 00 00 00 33  |.......2.▒.....3|

    000001a0  bf 3b 00 00 00 00 00 34  ed 67 00 00 00 00 00 35  |▒;.....4▒g.....5|

    000001b0  73 d0 00 00 00 00 00 36  6a 15 00 00 00 00 00 37  |s▒.....6j......7|

    000001c0  b4 bd 00 00 00 00 00 38  6f f2 00 00 00 00 00 39  |▒▒.....8o▒.....9|

    000001d0  b1 67 00 00 00 00 00 3a  26 03 00 00 00 00 00 3b  |▒g.....:&......;|

    000001e0  56 6c 00 00 00 00 00 3c  34 69 00 00 00 00 00 3d  |Vl.....<4i.....=|

    000001f0  b0 a9 00 00 00 00 00 3e  12 48 00 00 00 00 00 3f  |▒▒.....>.H.....?|

    00000200  a4 70 00 00 00 00 00 40  4c 43 00 00 00 00 00 41  |▒p.....@LC.....A|

    00000210  78 4d 00 00 00 00 00 42  fd 1a 00 00 00 00 00 43  |xM.....B▒......C|

    00000220  ad dd 00 00 00 00 00 44  e4 aa 00 00 00 00 00 45  |▒▒.....D▒.....E|

    00000230  b9 f5 00 00 00 00 00 46  7d 3f 00 00 00 00 00 47  |▒▒.....F}?.....G|

    00000240  23 b0 00 00 00 00 00 48  00 68 00 00 00 00 00 49  |#▒.....H.h.....I|

    00000250  a9 40 00 00 00 00 00 4a  f8 ba 00 00 00 00 00 4b  |▒@.....J▒.....K|

    00000260  aa 99 00 00 00 00 00 4c  3c b3 00 00 00 00 00 4d  |▒......L<▒.....M|

    00000270  6a 0a 00 00 00 00 00 4e  64 22 00 00 00 00 00 4f  |j......Nd".....O|

    00000280  22 46 00 00 00 00 00 50  97 dc 00 00 00 00 00 51  |"F.....P.▒.....Q|

                          

    7. Manually change either the data file or the protection file directly on the target machine (use 'vi' or any other tool to tweak the file content). Then try to read blocks from the initiator. The target should recognize a T10-DIF corruption, log that in the system log, report it back to the initiator where it will be reported as well.