2 Replies Latest reply on Apr 26, 2018 6:07 AM by halr

    Maximum message size supported for multicast?

    sriktib

      I was testing out the example multicast application provided by mellanox in section 7.3 of their document: http://www.mellanox.com/related-docs/prod_software/RDMA_Aware_Programming_user_manual.pdf . I'm able to compile and run the application and do multicast. I run using the following commands:

      Receiver side:

      example -M 224.0.0.1 -b 192.168.201.101 -c 1 -C1024 -S 1024

      ...

      Sender's side

      example  -M 224.0.0.1 -b 192.168.201.30 -c 1 -C1024 -S 1024 -s

       

      I believe this is the same as mckey utility.

       

      We have a requirement to transfer or copy 10-20 GB of data from server to multiple clients. I thought using multicast will be efficient. I have couple of questions about the given example (mckey):

       

      1. I'm able to only increase the message size to 4096 bytes. If I increase message size beyong that, there is a check for message_size as below:

      if (message_count && message_size > (1<< (port_attr.active_mtu + 7)))

       

      which kicks in and gives an error.

       

      The port attributes in our network are as follows:

       

      {state =IBV_PORT_ACTIVE, max_mtu = IBV_MTU_4096, active_mtu = IBV_MTU_4096, gid_tbl_len= 8, port_cap_flags = 642902090, max_msg_sz = 1073741824, bad_pkey_cntr = 0,  qkey_viol_cntr =0, pkey_tbl_len = 128, lid = 6, sm_lid = 9, lmc = 0 '\000', ax_vl_num = 3'\003', sm_sl = 0 '\000', subnet_timeout = 18 '\022', init_type_reply = 0'\000',  active_width = 2'\002', active_speed = 32 ' ', phys_state = 5 '\005', link_layer = 1 '\001',reserved = 0 '\000'}

       

      I can see that max_msg_sz supported by the port, from above, is 1GB and why am I not allowed to specify more than 4096 as message size?  In fact we have another application where QP type is RC and we are able to transfer messages up to 1GB there between the same nodes and HCA and port.  Is there a limitation while doing multicast?

       

      2. What is the maximum message count? Anythingbeyond 16K,  leads to error during rdma_create_qp. 

       

      Thanks

      Sri