4 Replies Latest reply on Dec 19, 2013 2:09 PM by startplus

    xenserver 6.2 LVMoISER

      Hi,

       

      Good News

      This is my current status :

      Done: load all required modules for iser transport

      Done: Creating storages type : LVM over iSCSI , but with transport iser , in XenCenter

       

      Only the transport was changed from tcp to iser .

      All XenServer features like:

      * Live Storage Migration

      * XenMotion ® live migration

      * VM Copy / Move

      are still available

       

      A test with a VM :

       

      win2008r2 - 1024MB Ram , 1CPU , Disk C, E , F ( XENSRC PVDisk SCSI Disk Device)

      Atto Disk Benchmark v2.47 default prefs , but Queue Depth : 10

       

      max iops read> 45000

      max iops write > 45000

      RMB / s> 1300

      WMB / s> 1300

       

      This is f : in Dom0

      [root @ xenserver ~ ] # iostat -x - m 1 sdc | grep sdc

      sdc0.00 0.00 0.00 45123.00 0.00 22.07 1.00 1.63 0.04 0.01 64.00

      sdc 0.00 0.00 45918.00 0.00 22.42 0.00 1.00 1.61 0.04 0.01 65.00

      see attached files

       

      # # # #Bad news :

       

      The server is not stable. At high I / O load , the server crashed .

      Here is the error message:

      <3 > [ 434.554641 ] iser : iser_start_rdma_unaligned_sg : Failed to allocate mem size 66 266240 for copying SGList

      <3 > [ 434.592026 ] iser : iser_prepare_write_cmd : Failed to register write cmd RDMA mem

      <3 > [ 434.629054 ] iser : iser_send_command : conn e8c3b0c8 failed task- > itt 83 err -12

      <3 > [ 434.648441 ] PCI - DMA: Out of SW- IOMMU space for 266240 bytes at device 0000:03:00.0

      <0 > [ 434.686043 ] ------------ [ cut here] ------------

      <2 > [ 434.704686 ] kernel BUG at / tmp/mlnx_iso.24095/OFED_topdir/BUILD/ofa_kernel-1.5.3/drivers/infiniband/ulp/iser/iser_memory.c : 98 !

      <0 > [ 434.743223 ] invalid opcode : 0000 [# 1 ] SMP<0 > [ 434.743224 ] load sysfs file: / sys/class/iscsi_session/session2/targetname

       

      Maybe someone an idea how I can solve the problem?

       

      more info in the file error.txt 

       


        • Re: xenserver 6.2 LVMoISER

          Which version of open-iscsi are you using?

           

          You would have had to graft in iser code into XS version of open-iscsi as this was stripped out by Citrix.

           

          You need to carefully match the open-iscsi version with the iser code base else you will get compatibility problems.

            • Re: xenserver 6.2 LVMoISER

              Open-iscsi version from xenserver6.2 install iso. The Iser interface option is part of it.

              [ XX] Loading iSCSI transport class v2.0-870.

              [ XX] iscsi: registered transport (iser)

              [ XX] iscsi: registered transport (tcp)

              dmesg looks good.

               

              ###

              <3>[ XX] iser: iser_start_rdma_unaligned_sg:Failed to allocate mem size 66 266240 for copying sglist

              <3>[ XX] iser: iser_prepare_write_cmd:Failed to register write cmd RDMA mem

              <3>[ XX] iser: iser_send_command:conn e8c3b0c8 failed task->itt 83 err -12

              <3>[ XX] PCI-DMA: Out of SW-IOMMU space for 266240 bytes at device 0000:03:00.0

               

              google "PCI-DMA: Out of SW-IOMMU space"

              * high I/O

               

              so i set  ... dom0 cmdline swiotlb=256

               

              [ XX] Software IO TLB enabled:

              [ XX] Aperture: 256 megabytes

              [ XX] Address size: 29 bits

              [ XX] Kernel range: c3976000 - d3976000

              [ XX] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)

              [ XX] PCI-DMA: mask is set to 38 bits

               

              and again a crash

               

              <0>[ 434.686043] ------------[ cut here ]------------

              <2>[ 434.704686] kernel BUG at /tmp/mlnx_iso.24095/OFED_topdir/BUILD/ofa_kernel-1.5.3/drivers/infiniband/ulp/iser/iser_memory.c:98!

              <0>[ 434.743223] invalid opcode: 0000 [#1] SMP

              <0>[ 434.743224] last sysfs file: /sys/class/iscsi_session/session2/targetname

               

              ###

              /**

              * iser_start_rdma_unaligned_sg

              */

              static int iser_start_rdma_unaligned_sg(struct iscsi_iser_task *iser_task,

                enum iser_data_dir cmd_dir)

              {

                int dma_nents;

                struct ib_device *dev;

                char *mem = NULL;

                struct iser_data_buf *data = &iser_task->data[cmd_dir];

                unsigned long cmd_data_len = data->data_len;

               

                if (cmd_data_len > ISER_KMALLOC_THRESHOLD)

                mem = (void *)__get_free_pages(GFP_ATOMIC,

                ilog2(roundup_pow_of_two(cmd_data_len)) - PAGE_SHIFT);

                else

                mem = kmalloc(cmd_data_len, GFP_ATOMIC);

               

                if (mem == NULL) {

                iser_err("Failed to allocate mem size %d %d for copying sglist\n",

                data->size,(int)cmd_data_len);

                return -ENOMEM;

                }

               

                if (cmd_dir == ISER_DIR_OUT) {

                /* copy the unaligned sg the buffer which is used for RDMA */

                struct scatterlist *sgl = (struct scatterlist *)data->buf;

                struct scatterlist *sg;

                int i;

                char *p, *from;

               

                p = mem;

                for_each_sg(sgl, sg, data->size, i) {

                from = kmap_atomic(sg_page(sg), KM_USER0);

                memcpy(p,

                from + sg->offset,

                sg->length);

                kunmap_atomic(from, KM_USER0);

                p += sg->length;

                }

                }

               

                sg_init_one(&iser_task->data_copy[cmd_dir].sg_single, mem, cmd_data_len);

                iser_task->data_copy[cmd_dir].buf =

                &iser_task->data_copy[cmd_dir].sg_single;

                iser_task->data_copy[cmd_dir].size = 1;

               

                iser_task->data_copy[cmd_dir].copy_buf = mem;

               

                dev = iser_task->iser_conn->ib_conn->device->ib_device;

                dma_nents = ib_dma_map_sg(dev,

                &iser_task->data_copy[cmd_dir].sg_single,

                1,

                (cmd_dir == ISER_DIR_OUT) ?

                DMA_TO_DEVICE : DMA_FROM_DEVICE);

               

               

              >>98 is here >> BUG_ON(dma_nents == 0);

               

               

                iser_task->data_copy[cmd_dir].dma_nents = dma_nents;

                return 0;

              }

               

              I see

               

              if (mem == NULL) {

                iser_err("Failed to allocate mem size %d %d for copying sglist\n",

                data->size,(int)cmd_data_len);

                return -ENOMEM;

                }

               

              result <3>[ XX] iser: iser_start_rdma_unaligned_sg:Failed to allocate mem size 66 266240 for copying sglist

               

              what can you do if the memory allocate failed?

              Perhabs wait and try again later?

               

              Preview Private Cloud meets ZFS on Infiniband

               

              preview.JPG.jpg

                • Re: xenserver 6.2 LVMoISER

                  Are you sure iser transport is created on target login? There is no iser modules within XS open-iscsi, so whilst the kernel modules loads, open-iscsi has no means to make a connection over iser.

                   

                  Can you try logging out of iscsi targets, and try from the command link to log into target like this.

                   

                  iscsiadm -m discovery -t st -p 10.0.0.1 -I iser

                  iscsiadm -m node -T  iqn.2010-10.com.example:storage-1000 -l

                   

                   

                  Let me know how it connects.

                    • Re: xenserver 6.2 LVMoISER

                      I'm at home and could not login from here.

                      By the way my english is bad... sorry.

                       

                      Openiscsi - iscsid, iscsiadm - base Xenserver install.

                      ....

                      !!!!! xenserverkernel6.2 added by script - save as tgz - extract to usb-disk !!!!!

                      Mount my usb disk and /mnt/mlnx..install - you know - i see errors and install bla and fasel with yum :-)

                      Another /mnt/mlnx...install - No errors - good

                      Rpm -i /mnt/RPM/kernel..xen.rpm

                      Vi /etc/infiniband/openibd.conf - yes/no - you know

                      Chkfg --level 2345 on opensm

                      Reboot

                      ....

                      Lsmod - mlnx*, ib_* - ok

                      Ibstat - ok

                      Ifconfig -a - i see ib0

                      Vi /etc/sysconfig/net...-scripts/chk-ib0 - static ip ...

                      Ifup ib0

                      ....

                      Storage is omnios - insert connectx card - ! opensm on linux/switch is running! config crossbow/comstar ...

                      ....

                      Xenserver console:

                      iscsiadm -m discover ... -I iser

                      iscsiadm -m node -L all

                      iscsiadm m session:

                      iser: .....

                       

                      I' m local in Berlin/Germany - so i say good night now....