5 Replies Latest reply on Oct 2, 2017 2:52 AM by inbusiness

    How to configure MCX354A-FCBT Mellanox InfiniBand speed at 56Gbps ?

    minouche

      Can someone help me to configure my MCX354A-FCBT Mellanox InfiniBand speed at 56Gbps. I have a MCX354A-FCBT Mellanox configured for InfiniBand but the speed remains at 40Gbps (All the components can speed at 56Gbps (card/Switch/Cable)! Thank a lot for your help. Here is my configuration:

       

      Operating System

      Fedora release 24 (Twenty Four)

      kernel 4.11.12-100.fc24.x86_64

       

      Mellanox card

      MCX354A-FCBT

       

      [root@aigle ~]# mlxconfig -d /dev/mst/mt4099_pci_cr0 q

       

      Device #1:

      ----------

      Device type:    ConnectX3      

      PCI device:     /dev/mst/mt4099_pci_cr0

       

      Configurations:                              Next Boot

               SRIOV_EN                            True(1)        

               NUM_OF_VFS                          8              

               LINK_TYPE_P1                        VPI(3)         

               LINK_TYPE_P2                        VPI(3)         

               LOG_BAR_SIZE                        3              

               BOOT_PKEY_P1                        0              

               BOOT_PKEY_P2                        0              

               BOOT_OPTION_ROM_EN_P1               True(1)        

               BOOT_VLAN_EN_P1                     False(0)       

               BOOT_RETRY_CNT_P1                   0              

               LEGACY_BOOT_PROTOCOL_P1             PXE(1)         

               BOOT_VLAN_P1                        1              

               BOOT_OPTION_ROM_EN_P2               True(1)        

               BOOT_VLAN_EN_P2                     False(0)       

               BOOT_RETRY_CNT_P2                   0              

               LEGACY_BOOT_PROTOCOL_P2             PXE(1)         

               BOOT_VLAN_P2                        1              

               IP_VER_P1                           IPv4(0)        

               IP_VER_P2                           IPv4(0)        

       

      [root@aigle ~]# mlxfwmanager --query

      Querying Mellanox devices firmware ...

       

      Device #1:

      ----------

        Device Type:      ConnectX3

        Part Number:      MCX354A-FCB_A2-A5

        Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6

        PSID:             MT_1090120019

        PCI Device Name:  /dev/mst/mt4099_pci_cr1

        Port1 GUID:       f45214030027f751

        Port2 GUID:       f45214030027f752

        Versions:         Current        Available    

           FW             2.42.5000      N/A          

           PXE            3.4.0752       N/A          

        Status:           No matching image found

       

      Device #2:

      ----------

        Device Type:      ConnectX3

        Part Number:      MCX354A-FCB_A2-A5

        Description:      ConnectX-3 VPI adapter card; dual-port QSFP; FDR IB (56Gb/s) and 40GigE; PCIe3.0 x8 8GT/s; RoHS R6

        PSID:             MT_1090120019

        PCI Device Name:  /dev/mst/mt4099_pci_cr0

        Port1 GUID:       0002c9030032e311

        Port2 GUID:       0002c9030032e312

        Versions:         Current        Available    

           FW             2.42.5000      N/A          

           PXE            3.4.0752       N/A          

        Status:           No matching image found

       

      [root@aigle ~]# ibstat

      CA 'mlx4_0'

      CA type: MT4099

      Number of ports: 2

      Firmware version: 2.42.5000

      Hardware version: 1

      Node GUID: 0x0002c9030032e310

      System image GUID: 0x0002c9030032e313

      Port 1:

      State: Active

      Physical state: LinkUp

      Rate: 40 (FDR10)

      Base lid: 6

      LMC: 0

      SM lid: 1

      Capability mask: 0x02514868

      Port GUID: 0x0002c9030032e311

      Link layer: InfiniBand

      Port 2:

      State: Active

      Physical state: LinkUp

      Rate: 40 (FDR10)

      Base lid: 7

      LMC: 0

      SM lid: 1

      Capability mask: 0x02514868

      Port GUID: 0x0002c9030032e312

      Link layer: InfiniBand

      CA 'mlx4_1'

      CA type: MT4099

      Number of ports: 2

      Firmware version: 2.42.5000

      Hardware version: 1

      Node GUID: 0xf45214030027f750

      System image GUID: 0xf45214030027f753

      Port 1:

      State: Active

      Physical state: LinkUp

      Rate: 40 (FDR10)

      Base lid: 8

      LMC: 0

      SM lid: 1

      Capability mask: 0x02514868

      Port GUID: 0xf45214030027f751

      Link layer: InfiniBand

      Port 2:

      State: Active

      Physical state: LinkUp

      Rate: 40 (FDR10)

      Base lid: 9

      LMC: 0

      SM lid: 1

      Capability mask: 0x02514868

      Port GUID: 0xf45214030027f752

      Link layer: InfiniBand

       

      [root@aigle ~]# ibv_devinfo -v

      hca_id: mlx4_0

      transport: InfiniBand (0)

      fw_ver: 2.42.5000

      node_guid: 0002:c903:0032:e310

      sys_image_guid: 0002:c903:0032:e313

      vendor_id: 0x02c9

      vendor_part_id: 4099

      hw_ver: 0x1

      board_id: MT_1090120019

      phys_port_cnt: 2

      max_mr_size: 0xffffffffffffffff

      page_size_cap: 0xfffffe00

      max_qp: 393144

      max_qp_wr: 16351

      device_cap_flags: 0x057e9c76

      BAD_PKEY_CNTR

      BAD_QKEY_CNTR

      AUTO_PATH_MIG

      CHANGE_PHY_PORT

      UD_AV_PORT_ENFORCE

      PORT_ACTIVE_EVENT

      SYS_IMAGE_GUID

      RC_RNR_NAK_GEN

      XRC

      Unknown flags: 0x056e8000

      device_cap_exp_flags: 0x5000401600000000

      EXP_DEVICE_QPG

      EXP_UD_RSS

      EXP_CROSS_CHANNEL

      EXP_MR_ALLOCATE

      EXT_ATOMICS

      EXP_MASKED_ATOMICS

      max_sge: 32

      max_sge_rd: 30

      max_cq: 65408

      max_cqe: 4194303

      max_mr: 524032

      max_pd: 32764

      max_qp_rd_atom: 16

      max_ee_rd_atom: 0

      max_res_rd_atom: 6290304

      max_qp_init_rd_atom: 128

      max_ee_init_rd_atom: 0

      atomic_cap: ATOMIC_HCA (1)

      log atomic arg sizes (mask) 0x8

      masked_log_atomic_arg_sizes (mask) 0x8

      masked_log_atomic_arg_sizes_network_endianness (mask) 0x0

      max fetch and add bit boundary 64

      log max atomic inline 3

      max_ee: 0

      max_rdd: 0

      max_mw: 0

      max_raw_ipv6_qp: 0

      max_raw_ethy_qp: 0

      max_mcast_grp: 131072

      max_mcast_qp_attach: 244

      max_total_mcast_qp_attach: 31981568

      max_ah: 2147483647

      max_fmr: 0

      max_srq: 65472

      max_srq_wr: 16383

      max_srq_sge: 31

      max_pkeys: 128

      local_ca_ack_delay: 15

      hca_core_clock: 427000

      max_klm_list_size: 0

      max_send_wqe_inline_klms: 0

      max_umr_recursion_depth: 0

      max_umr_stride_dimension: 0

      general_odp_caps:

      max_size: 0x0

      rc_odp_caps:

      NO SUPPORT

      uc_odp_caps:

      NO SUPPORT

      ud_odp_caps:

      NO SUPPORT

      dc_odp_caps:

      NO SUPPORT

      xrc_odp_caps:

      NO SUPPORT

      raw_eth_odp_caps:

      NO SUPPORT

      max_dct: 0

      max_device_ctx: 1016

      Multi-Packet RQ is not supported

      rx_pad_end_addr_align: 0

      tso_caps:

      max_tso: 0

      packet_pacing_caps:

      qp_rate_limit_min: 0kbps

      qp_rate_limit_max: 0kbps

      ooo_caps:

      ooo_rc_caps  = 0x0

      ooo_xrc_caps = 0x0

      ooo_dc_caps  = 0x0

      ooo_ud_caps  = 0x0

      sw_parsing_caps:

      supported_qp:

      tag matching not supported

      Device ports:

      port: 1

      state: PORT_ACTIVE (4)

      max_mtu: 4096 (5)

      active_mtu: 4096 (5)

      sm_lid: 1

      port_lid: 6

      port_lmc: 0x00

      link_layer: InfiniBand

      max_msg_sz: 0x40000000

      port_cap_flags: 0x02514868

      max_vl_num: 8 (4)

      bad_pkey_cntr: 0x0

      qkey_viol_cntr: 0x0

      sm_sl: 0

      pkey_tbl_len: 128

      gid_tbl_len: 128

      subnet_timeout: 18

      init_type_reply: 0

      active_width: 4X (2)

      active_speed: 10.0 Gbps (8)

      phys_state: LINK_UP (5)

      GID[  0]: fe80:0000:0000:0000:0002:c903:0032:e311

      port: 2

      state: PORT_ACTIVE (4)

      max_mtu: 4096 (5)

      active_mtu: 4096 (5)

      sm_lid: 1

      port_lid: 7

      port_lmc: 0x00

      link_layer: InfiniBand

      max_msg_sz: 0x40000000

      port_cap_flags: 0x02514868

      max_vl_num: 8 (4)

      bad_pkey_cntr: 0x0

      qkey_viol_cntr: 0x0

      sm_sl: 0

      pkey_tbl_len: 128

      gid_tbl_len: 128

      subnet_timeout: 18

      init_type_reply: 0

      active_width: 4X (2)

      active_speed: 10.0 Gbps (8)

      phys_state: LINK_UP (5)

      GID[  0]: fe80:0000:0000:0000:0002:c903:0032:e312

       

       

      Switch

      Part Info

      --------

      Type: SX6036

      S/N: IL23190198

      P/N: 712498-B21

      Chassis system GUID: 00:02:C9:03:00:AC:6C:20

      Asic FW version: 9.4.3580

      LID: 1

      Node GUID: 00:02:C9:03:00:AC:6C

       

      Installed MLNX-OS Images

      ------------------------

      Partition 1 - Active Image (partition of next boot)

      PPC_M460EX 3.6.4006 2017-07-03 16:17:35 ppc

       

      Partition 2

      PPC_M460EX 3.6.3004 2017-02-05 17:31:50 ppc

       

      Port Info

      ----------

      Port number : 1

      Port type : IB

      IB Subnet : infiniband-default

      Port description :

      Logical port state :    Active

      Physical port state : LinkUp

      Current line rate : 40.0 Gbps

      Supported speeds : sdr, ddr, qdr, fdr10

      Speed :         fdr10

      Supported widths : 1X, 4X

      Width :         4X

      Max supported MTUs : 4096

      MTU :         4096

      VL capabilities : VL0 - VL7

      Operational VLs : VL0 - VL7

      Supported LLR speeds : FDR10, FDR

      LLR Status : Active

       

      Transceiver Information 

      Identifier :                 QSFP+        

      Cable/ Module type : Passive copper, unequalized

      Infiniband speeds : SDR , DDR , QDR , FDR

      Vendor :         Mellanox

      Cable length :         2 m

      Part number :         MC2207130-002

      Revision :         A3

      Serial number :         MT1710VS05863

       

      Subnet Manager (SM) Status

      --------------------------

      SM Status

      Local SM running 1 hour 26 minutes 52 seconds

      SM Priority 7 State running

      Failures 0 Autostart true

      Routing Engine Used minhop

      SM version OpenSM4.7.0.MLNX20170511.3016205

        • Re:  How to configure MCX354A-FCBT Mellanox InfiniBand speed at 56Gbps ?
          inbusiness

          Hi!

          You have SX6036T FDR10 switch.

          Your switch's port support up to FDR10 (40Gb, 64/66bit encording).

          If you want FDR14(56Gb) speed, you must change your switch to SX6036F or SX6036G.

           

          Best Regard,

          Jae-Hoon Choi

            • Re:  How to configure MCX354A-FCBT Mellanox InfiniBand speed at 56Gbps ?
              aviap

              From the output prints you have presented, it looks like your SX6036 switch should be good & supports 56Gb (FDR)

              - cables & nics are also fine and capable of fdr

              - cables present: Infiniband speeds : SDR , DDR , QDR , FDR

              - switch presents:

              Supported LLR speeds : FDR10, FDR - which indicates you can set it to 56Gb

              Supported speeds : sdr, ddr, qdr, fdr10   - here you see that fdr is missing

              so, in my view - all you have to do is to run the following command on the switch that will add FDR on "supported speeds" list

              run on CLI commnd:

              (config) # interface ib <interface#> speed sdr ddr qdr fdr10 fdr force

              (config) # configure write (to save the changes)


              you should now see: Supported speeds : sdr, ddr, qdr, fdr10, fdr

              this should enable you to uses 56Gb on the switch and on the nics as well

                • Re:  How to configure MCX354A-FCBT Mellanox InfiniBand speed at 56Gbps ?
                  inbusiness

                   

                  HPE Part Number 712498-B21 is SX6036T FDR10 switch.

                  Therefore in this case user can't use FDR14 56Gb speed in fabric.

                  HPE Support document - HPE Support Center

                   

                   

                  Why did you give a incorrect information to community users?

                   

                  Best Regard,

                  Jae-Hoon Choi

                    • Re:  How to configure MCX354A-FCBT Mellanox InfiniBand speed at 56Gbps ?
                      minouche

                      Hi Jae-Hoon, Hi  aviap

                       

                      I'm sorry to be late. Thank a lot for your answers. Now, it seems clear that I can not use FDR because my switch is not compatible. I still tested to force FDR but "% the requested speed value is not allowed for this port":

                       

                      Mellanox-SX6036 [standalone: master] # configure terminal

                      Mellanox-SX6036 [standalone: master] (config) #

                      Mellanox-SX6036 [standalone: master] (config) # show running-config

                      ##llanox-SX6036 [standalone: master] (config) # show running-config

                      ## Running database "initial"

                      ## Generated at 2017/10/01 20:10:51 +0200

                      ## Hostname: Mellanox-SX6036

                      ##

                       

                       

                      ##

                      ## Running-config temporary prefix mode setting

                      ##

                      no cli default prefix-modes enable

                       

                       

                      ##

                      ## Network interface configuration

                      ##

                         interface ib 1/1 mtu 4K

                         interface ib 1/1 op-vls 8

                         interface ib 1/1 speed sdr ddr qdr fdr10

                         interface ib 1/1 width 5

                      no interface mgmt0 dhcp

                         interface mgmt0 ip address 192.168.0.252 /24

                        

                      ##

                      ## Network interface IPv6 configuration

                      ##

                      no interface mgmt0 ipv6 address autoconfig default

                      no interface mgmt0 ipv6 enable

                        

                      ...

                       

                       

                      Mellanox-SX6036 [standalone: master] (config) # show interface ib status

                       

                       

                      Interface      Description                                IB Subnet            Speed           Current line rate   Logical port state   Physical port state  

                      ---------      -----------                                ------------------   ---------       -----------------   ------------------   -------------------  

                      IB1/1                                                     infiniband-default   fdr10           40.0 Gbps           Active               LinkUp               

                      IB1/2                                                     infiniband-default   fdr10           40.0 Gbps           Active               LinkUp               

                      IB1/3                                                     infiniband-default   fdr10           40.0 Gbps           Active               LinkUp               

                      IB1/4                                                     infiniband-default   fdr10           40.0 Gbps           Active               LinkUp               

                      IB1/5                                                     infiniband-default   -               -                   Down                 Polling              

                      IB1/6                                                     infiniband-default   -               -                   Down                 Polling              

                      IB1/7                                                     infiniband-default   -               -                   Down                 Polling              

                      IB1/8                                                     infiniband-default   -               -                   Down                 Polling              

                      IB1/9                                                     infiniband-default   -               -                   Down                 Polling              

                      IB1/10                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/11                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/12                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/13                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/14                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/15                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/16                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/17                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/18                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/19                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/20                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/21                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/22                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/23                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/24                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/25                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/26                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/27                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/28                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/29                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/30                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/31                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/32                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/33                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/34                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/35                                                    infiniband-default   -               -                   Down                 Polling              

                      IB1/36                                                    infiniband-default   -               -                   Down                 Polling              

                       

                       

                      Mellanox-SX6036 [standalone: master] (config) # interface ib 1/1 speed sdr ddr qdr fdr10 fdr force

                      % the requested speed value is not allowed for this port

                      Mellanox-SX6036 [standalone: master] (config) #

                       

                      No matter if the switch is only FDR10 and not FDR it is just for educational purpose, I’ m learning InfiniBand technology. Thank a lot.

                       

                      Best Regard