HowTo Configure TGT enabled with iSER Transport for RHEL

Version 21

    This post shows how to configure set tgt ISCSI client and target to work in iSER transport mode for RHEL/Fedora/CentOS distribution.

     

     

    References

     

     

    Setup

     

    Network

    The connectivity between the iSCSI client and target should be on lossless network required for RoCE. PFC or global pause should be enabled across the network.

     

    Notes:

    (1) MLNX_OFED installation install by default all relevant modules to enable iSER.

    (2) For servers without MLNX_OFED, you need to install several group modules using the commands

    #yum groupinstall "InfiniBand Support"

                


    (3) The only configuration change between regular iSCSI and iSER is a simple flag  More information on the commands can be found on the man pages of the application.

    (4) The basic management utility is ‘iscsiadm’, which is part of iSCSI package on any distro, and is capable of configuring iSER.

     

    iSCSI target (Server) configuration:

    Server configuration is generic and fairly simple. In the following link you can find example for RHEL, Fedora or CentOS

    http://www.switchroot.com/how-to-configure-a-iscsi-target-iscsi-server-on-rhelfedora-or-centos

     

    iSCSI client configuration:

    Client configuration (Initiator) is generic and fairly simple as well. In the following link you can find example for RHEL, Fedora or CentOS

    http://www.switchroot.com/how-to-configure-a-iscsi-initiator-iscsi-client-on-rhelfedora-or-centos

     

    Server configuration example:

    #yum install -y scsi-target-utils

    #tgtd

    #tgt-setup-lun -n tg0 -d /tmp/null -b null -t iser

    Creating new target (name=iqn.2001-04.com.my-server-tg0, tid=1)

    Adding a logical unit (/tmp/null) to target, tid=1

    Setting backing store type: rdwr

    Accepting connections from all initiators

         

     

    Client configuration example:

    # yum install -y lsscsi fio iscsi-initiator-utils

    # modprobe ib_iser

    #iscsiadm -m discovery --op=new --op=delete --type sendtargets --portal 11.11.10.1:3260  -I iser

    11.11.10.1:3260,1 iqn.2001-04.com.reg-r-vrt-010-tg0

    # iscsiadm -m node -l

    Logging in to [iface: iser, target: iqn.2001-04.com.reg-r-vrt-010-tg0, portal: 11.11.1.10,3260] (multiple)

    Login to [iface: iser, target: iqn.2001-04.com.reg-r-vrt-010-tg0, portal: 11.11.1.10,3260] successful.

    # fio --rw=randread --bs=64k --numjobs=4 --iodepth=8 --runtime=30 --time_based --loops=1 --ioengine=libaio --direct=1 --invalidate=1 --fsync_on_close=1 --randrepeat=1 --norandommap --exitall --name task1 --filename=/dev/sdb

    sync_on_close=1 --randrepeat=1 --norandommap --exitall --name task1 --filename=/dev/sdb

    task1: (g=0): rw=randread, bs=64K-64K/64K-64K/64K-64K, ioengine=libaio, iodepth=8

    ...

    fio-2.1.7

    Starting 4 processes

    Jobs: 4 (f=4): [rrrr] [100.0% done] [4304MB/0KB/0KB /s] [68.9K/0/0 iops] [eta 00m:00s]

    ...

         

     

    For doing discovery over iser both the initiator and target need to support it. In the initiator side both the kernel driver and the user-space open-iscsi package needs to support it.

    If there is a support problem then you can do discovery over tcp and then update the connection to be iser for the login.

     

    # iscsiadm -m discovery --op=new --op=delete --type sendtargets --portal 11.11.10.1:3260

    # iscsiadm -m node -T <target-name> -o update -n iface.transport_name -v iser

     

    Troubleshooting

    1. if "yum install -y scsi-target-utils" command doesn't work (e.g. in case of CenOS 7) you will need to manually download and compile the package.

     

    For example:

    # cd /tmp/

    # yum install -y librdmacm-devel libibverbs-devel libaio-devel

    # yum install ftp://fr2.rpmfind.net/linux/epel/7/x86_64/p/perl-Config-General-2.51-2.el7.noarch.rpm

    # git clone git://github.com/fujita/tgt.git

    # cd tgt/

    # make rpm

    # yum install /tmp/tgt/pkg/RPMS/x86_64/scsi-target-utils-1.0.60-v1.0.60.x86_64.rpm

    # tgtd -h

    Linux SCSI Target framework daemon, version 1.0.60

     

    Usage: tgtd [OPTION]

    -f, --foreground        make the program run in the foreground

    -C, --control-port NNNN use port NNNN for the mgmt channel

    -t, --nr_iothreads NNNN specify the number of I/O threads

    -d, --debug debuglevel  print debugging information

    -V, --version           print version and exit

    -h, --help              display this help and exit

     

    2. Verify that the client is connected with the ISER transport. if not, it may point on a configuration error.

    # iscsiadm -m node -o show | grep iser

    iface.iscsi_ifacename = iser

    iface.transport_name = iser