7 Replies Latest reply on May 31, 2013 12:45 PM by justinclift

    Infiniband on Linux... defaults to datagram mode somehow?

      RHEL v5.5

      kernel 2.6.18-194.el5

       

      What I have discovered, is that if you don't have "ONBOOT=yes" in your ifcfg-ib# script... the interface will come up in "datagram" mode, regardless of what is in /etc/infiniband/openib.conf.   Can anyone tell me why this happens or if its by design even?

       

       

      cat openib.conf

      # Start HCA driver upon boot

      ONBOOT=yes

       

       

      # Node description

      NODE_DESC=$(hostname -s)

       

       

      # Wait for NODE_DESC_TIME_BEFORE_UPDATE sec before node_desc update

      NODE_DESC_TIME_BEFORE_UPDATE=10

       

       

      # Load UCM module

      UCM_LOAD=no

       

       

      # Load RDMA_CM module

      RDMA_CM_LOAD=yes

       

       

      # Load RDMA_UCM module

      RDMA_UCM_LOAD=yes

       

       

      # Increase ib_mad thread priority

      RENICE_IB_MAD=no

       

       

      # Run sysctl performance tuning script

      RUN_SYSCTL=yes

       

       

      # Load MTHCA

      MTHCA_LOAD=yes

       

       

      # Load QIB

      QIB_LOAD=yes

       

       

      # Load MLX4 modules

      MLX4_LOAD=yes

       

       

      # Load MLX4_EN module

      MLX4_EN_LOAD=yes

       

       

      # Start FC service

      MLX4_FC_LOAD=no

       

       

      # Load MLX4_VNIC module

      MLX4_VNIC_LOAD=no

       

       

      # Load CXGB3 modules

      CXGB3_LOAD=yes

       

       

      # Load NES modules

      NES_LOAD=yes

       

       

      # Load IPoIB

      IPOIB_LOAD=yes

       

       

      # Enable IPoIB Connected Mode

      SET_IPOIB_CM=yes

       

       

      # Load SDP module

      SDP_LOAD=no

       

       

      # Load SRP module

      SRP_LOAD=no

       

       

      # Load ISER module

      ISER_LOAD=no

       

       

      # Load RDS module

      RDS_LOAD=no

       

       

      # Enable SRP High Availability daemon

      SRPHA_ENABLE=no

      SRP_DAEMON_ENABLE=no

        • Re: Infiniband on Linux... defaults to datagram mode somehow?
          ali

          Are you using MLNX OFED 2.0?

          Starting from MLNX OFED 2.0, the default mode is UD.

          From the User Manual section 1.3.4:

          The IP over IB (IPoIB) driver is a network interface implementation over InfiniBand. IPoIB

          encapsulates IP datagrams over an InfiniBand connected or datagram transport service. IPoIB

          pre-appends the IP datagrams with an encapsulation header, and sends the outcome over the

          InfiniBand transport service. The transport service is Unreliable Datagram (UD) by default, but it

          may also be configured to be Reliable Connected (RC). The interface supports unicast, multicast

          and broadcast. For details, see Chapter 4.3, “IP over InfiniBand”.

           

          IPoIB UD performance was significantly improved in MLNX OFED 2.0, thus it was set to be the default mode.

          You can still use IPoIB CM mode if you wish.

           

          This still doesn't explain why the config in openibd.conf is not picked up, I suggest you check /etc/init.d/openibd to see if the config file is parsed as expected.

            • Re: Infiniband on Linux... defaults to datagram mode somehow?

              Well, I think its just what comes with the kernel source...

               

              ################################################################################

               

               

              module ib_ipoib

               

               

              filename:       /lib/modules/2.6.18-194.el5/updates/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko

              license:        Dual BSD/GPL

              description:    IP-over-InfiniBand net driver

              author:         Roland Dreier

              srcversion:     22D6DA2213921B790D1BD7D

              depends:        ib_cm,ipv6,ib_core,ib_core,ib_sa,ipoib_helper

              vermagic:       2.6.18-194.el5 SMP mod_unload gcc-4.1

              parm:           max_nonsrq_conn_qp:Max number of connected-mode QPs per interface (applied only if shared receive queue is not available) (int)

              parm:           mcast_debug_level:Enable multicast debug tracing if > 0 (int)

              parm:           send_queue_size:Number of descriptors in send queue (int)

              parm:           recv_queue_size:Number of descriptors in receive queue (int)

              parm:           lro:Enable LRO (Large Receive Offload) (bool)

              parm:           lro_max_aggr:LRO: Max packets to be aggregated (default = 64) (int)

              parm:           debug_level:Enable debug tracing if > 0 (int)

              parm:           mc_sendonly_timeout:Enable debug tracing if > 0 (int)

               

              ################################################################################

               

               

              module ib_cm

               

               

              filename:       /lib/modules/2.6.18-194.el5/updates/kernel/drivers/infiniband/core/ib_cm.ko

              license:        Dual BSD/GPL

              description:    InfiniBand CM

              author:         Sean Hefty

              srcversion:     08CBDB36AB5BDE6F83763F9

              depends:        ib_core,ib_mad,ib_sa,ib_core

              vermagic:       2.6.18-194.el5 SMP mod_unload gcc-4.1

              parm:           max_timeout:Maximum IB CM per message timeout (default=21, or ~8 seconds) (int)

               

               

              ################################################################################

               

              This is an HP Ibrix storage server, and its sort of "locked down" as far as packages go.  I was just curious if anyone had an idea as to why it was falling to UD mode. Thanks for any help you guys can provide.