0 Replies Latest reply on Jul 24, 2015 8:59 PM by hmli

    mlx4_core 0000:81:00.0: vpd r/w failed. MPI startup(): ofa fabric is not available and fallback fabric is not enabled

    hmli

      After running some MPI jobs, the job will exit with error:

      [69] MPI startup(): ofa fabric is not available and fallback fabric is not enabled

      The dmesg show:

      mlx4_core 0000:81:00.0: vpd r/w failed.  This is likely a firmware bug on this device.  Contact the card vendor for a firmware update.

      mlx4_core 0000:81:00.0: vpd r/w failed.  This is likely a firmware bug on this device.  Contact the card vendor for a firmware update.

      LustreError: 28769:0:(file.c:186:ll_close_inode_openhandle()) tc4600fs-clilmv-ffff8810745d2c00: inode [0x2000015fb:0x1f8f0:0x0] mdc close failed: rc = -13

      LustreError: 28752:0:(file.c:186:ll_close_inode_openhandle()) tc4600fs-clilmv-ffff8810745d2c00: inode [0x2000015fb:0x1f8f2:0x0] mdc close failed: rc = -13

      LustreError: 28752:0:(file.c:186:ll_close_inode_openhandle()) Skipped 1 previous similar message

      OS:  CentOS release 6.5  2.6.32-431.29.2.el6.x86_64

       

      mstflint -d 0000:81:00.0 query:

      Image type: FS2

      FW Version: 2.30.8000

      Device ID: 4099

      Description: Node Port1 Port2 Sys image

      GUIDs: 0000e8611f104684 0000e8611f104685 0000e8611f104686 0000e8611f104687

      MACs: e8611f104684 e8611f104685

      VSD:

      PSID: MT_1100110019

       

      ibv_devinfo:

      hca_id:    mlx4_0

          transport:            InfiniBand (0)

          fw_ver:                2.30.8000

          node_guid:            0000:e861:1f10:4684

          sys_image_guid:            0000:e861:1f10:4687

          vendor_id:            0x02c9

          vendor_part_id:            4099

          hw_ver:                0x1

          board_id:            MT_1100110019

          phys_port_cnt:            1

              port:    1

                  state:            PORT_ACTIVE (4)

                  max_mtu:        4096 (5)

                  active_mtu:        4096 (5)

                  sm_lid:            80

                  port_lid:        149

                  port_lmc:        0x00

                  link_layer:        InfiniBand

      ibqueryerrors

      Errors for "node11 HCA-1"

         GUID 0x6c025792bf00374d port 1: [PortXmitWait == 428]

      Errors for "node12 HCA-1"

         GUID 0x6c025792bf0052f9 port 1: [PortXmitWait == 4034525]

      Errors for "node13 HCA-1"

         GUID 0x6c025792bf005029 port 1: [PortXmitWait == 1332591]

      Errors for "node14 HCA-1"

         GUID 0x6c025792bf00378d port 1: [PortXmitWait == 497142]

      Errors for "node15 HCA-1"

         GUID 0x6c025792bf003769 port 1: [PortXmitWait == 13]

      Errors for "node16 HCA-1"

         GUID 0x6c025792bf003795 port 1: [PortXmitWait == 467501]

      Errors for "node17 HCA-1"

         GUID 0x6c025792bf005349 port 1: [PortXmitWait == 24680]

      Errors for "node18 HCA-1"

         GUID 0x6c025792bf005365 port 1: [PortXmitWait == 293558]

      Errors for "node19 HCA-1"

         GUID 0x6c025792bf003799 port 1: [PortXmitWait == 3864438]

      Errors for "node20 HCA-1"

         GUID 0x6c025792bf003751 port 1: [PortXmitWait == 8791370]

      Errors for "node36 HCA-1"

         GUID 0x46d2c92000002371 port 1: [PortXmitWait == 505251]

      Errors for "node25 HCA-1"

         GUID 0x2c903001f7b01 port 1: [PortXmitWait == 22433658]

      Errors for "node24 HCA-1"

         GUID 0x2c903001f7011 port 1: [PortXmitWait == 231280]

      Errors for "node37 HCA-1"

         GUID 0x46d2c92000004ad1 port 1: [PortXmitWait == 35467]

      Errors for "node33 HCA-1"

         GUID 0x46d2c920000052b1 port 1: [PortXmitWait == 7150432]

      Errors for "node21 HCA-1"

         GUID 0x2c903001f8641 port 1: [PortXmitWait == 42722907]

      Errors for "node31 HCA-1"

         GUID 0x46d2c92000012201 port 1: [PortXmitWait == 21578]

      Errors for "node39 HCA-1"

         GUID 0x46d2c920000050c1 port 1: [PortXmitWait == 8356]

      Errors for "node34 HCA-1"

         GUID 0x46d2c920000058a1 port 1: [PortXmitWait == 58677393]

      Errors for "node35 HCA-1"

         GUID 0x46d2c920000049d1 port 1: [PortXmitWait == 45396994]

      Errors for "node38 HCA-1"

         GUID 0x46d2c920000052a1 port 1: [PortXmitWait == 7981989]

      Errors for "node40 HCA-1"

         GUID 0x46d2c92000005401 port 1: [PortXmitWait == 92006320]

      Errors for "node41 HCA-1"

         GUID 0x46d2c92000005121 port 1: [PortXmitWait == 10843583]

      Errors for "node42 HCA-1"

         GUID 0x46d2c92000005851 port 1: [PortXmitWait == 6418548]

      Errors for "node43 HCA-1"

         GUID 0x46d2c92000005441 port 1: [PortXmitWait == 2123159]

      Errors for "node44 HCA-1"

         GUID 0x46d2c920000053f1 port 1: [PortXmitWait == 15341643]

      Errors for "node32 HCA-1"

         GUID 0x46d2c9200000c971 port 1: [PortXmitWait == 102666977]

      Errors for "node29 HCA-1"

         GUID 0x2c903001f8601 port 1: [PortXmitWait == 6830999]

      Errors for "node28 HCA-1"

         GUID 0x2c903001f64b1 port 1: [PortXmitWait == 79086076]

      Errors for "node22 HCA-1"

         GUID 0x2c903001f8661 port 1: [PortXmitWait == 5017]

      Errors for "node23 HCA-1"

         GUID 0x2c903001f6f01 port 1: [PortXmitWait == 37801]

      Errors for "node30 HCA-1"

         GUID 0x2c903001f8651 port 1: [PortXmitWait == 97280452]

      Errors for "node27 HCA-1"

         GUID 0x2c903001f5641 port 1: [PortXmitWait == 126472513]

      Errors for 0x6c025792bf003668 "Infiniscale-IV Mellanox Technologies"

         GUID 0x6c025792bf003668 port ALL: [SymbolErrorCounter == 52034] [LinkErrorRecoveryCounter == 18] [PortRcvErrors == 52089] [PortRcvSwitchRelayErrors == 6233] [PortXmitDiscards == 13] [PortXmitWait == 100747894]

         GUID 0x6c025792bf003668 port 1: [PortRcvErrors == 1] [PortXmitDiscards == 1] [PortXmitWait == 4294967295]

         GUID 0x6c025792bf003668 port 2: [PortXmitWait == 4294967295]

         GUID 0x6c025792bf003668 port 3: [PortXmitWait == 4294967295]

         GUID 0x6c025792bf003668 port 4: [PortXmitWait == 4294967295]

         GUID 0x6c025792bf003668 port 5: [PortXmitWait == 12550054]

         GUID 0x6c025792bf003668 port 6: [PortXmitDiscards == 1] [PortXmitWait == 4294967295]

         GUID 0x6c025792bf003668 port 7: [PortXmitWait == 3662292368]

         GUID 0x6c025792bf003668 port 8: [PortXmitWait == 4294967295]

         GUID 0x6c025792bf003668 port 9: [PortXmitWait == 4294967295]

         GUID 0x6c025792bf003668 port 10: [PortXmitWait == 4294967295]

         GUID 0x6c025792bf003668 port 17: [LinkErrorRecoveryCounter == 2] [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 2] [PortXmitWait == 4294967295]

         GUID 0x6c025792bf003668 port 19: [LinkErrorRecoveryCounter == 2] [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 1] [PortXmitWait == 4294967295]

         GUID 0x6c025792bf003668 port 21: [SymbolErrorCounter == 52034] [LinkErrorRecoveryCounter == 2] [PortRcvErrors == 51880] [PortRcvSwitchRelayErrors == 2051] [PortXmitDiscards == 1] [PortXmitWait == 4294967295]

         GUID 0x6c025792bf003668 port 23: [LinkErrorRecoveryCounter == 2] [PortRcvSwitchRelayErrors == 169] [PortXmitDiscards == 1] [PortXmitWait == 4294967295]

         GUID 0x6c025792bf003668 port 25: [PortRcvSwitchRelayErrors == 29] [PortXmitDiscards == 1] [PortXmitWait == 4294967295]

         GUID 0x6c025792bf003668 port 27: [LinkErrorRecoveryCounter == 2] [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 1] [PortXmitWait == 4294967295]

         GUID 0x6c025792bf003668 port 29: [LinkErrorRecoveryCounter == 2] [PortRcvErrors == 208] [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 2] [PortXmitWait == 4294967295]

         GUID 0x6c025792bf003668 port 31: [LinkErrorRecoveryCounter == 2] [PortRcvSwitchRelayErrors == 3292] [PortXmitDiscards == 2] [PortXmitWait == 4294967295]

         GUID 0x6c025792bf003668 port 33: [LinkErrorRecoveryCounter == 2] [PortRcvSwitchRelayErrors == 687] [PortXmitWait == 720872785]

         GUID 0x6c025792bf003668 port 35: [LinkErrorRecoveryCounter == 2] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

      Errors for "node103 HCA-1"

         GUID 0xe8611f104a35 port 1: [PortXmitWait == 961]

      Errors for "node110 HCA-1"

         GUID 0xe8611f104af5 port 1: [PortXmitWait == 1183983]

      Errors for "node102 HCA-1"

         GUID 0xe8611f104865 port 1: [PortXmitWait == 1714217]

      Errors for "node109 HCA-1"

         GUID 0xe8611f1047b5 port 1: [PortXmitWait == 2812689]

      Errors for "io1 HCA-1"

         GUID 0x2c90300418f61 port 1: [PortXmitWait == 39237]

      Errors for "node2 HCA-1"

         GUID 0x2c903001ba811 port 1: [PortXmitWait == 9660175]

      Errors for "node1 HCA-1"

         GUID 0x2c903001ba381 port 1: [PortXmitWait == 51238]

      Errors for "node9 HCA-1"

         GUID 0x2c90300419061 port 1: [PortXmitWait == 25329]

      Errors for "node8 HCA-1"

         GUID 0x2c90300418f71 port 1: [PortXmitWait == 121]

      Errors for "node108 HCA-1"

         GUID 0xe8611f104795 port 1: [PortXmitWait == 1111446]

      Errors for "node107 HCA-1"

         GUID 0xe8611f104a15 port 1: [PortXmitWait == 1889]

      Errors for "node4 HCA-1"

         GUID 0x2c903001ba7f1 port 1: [PortXmitWait == 26625247]

      Errors for 0x2c903008ccf00 "SwitchX -  Mellanox Technologies"

         GUID 0x2c903008ccf00 port ALL: [LinkDownedCounter == 88] [PortRcvRemotePhysicalErrors == 16231] [PortRcvSwitchRelayErrors == 157] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 1: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 2: [PortRcvSwitchRelayErrors == 129] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 6: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 7: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 8: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 9: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 10: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 12: [LinkDownedCounter == 9] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 13: [LinkDownedCounter == 8] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 14: [LinkDownedCounter == 5] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 15: [LinkDownedCounter == 5] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 16: [LinkDownedCounter == 6] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 17: [LinkDownedCounter == 6] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 19: [LinkDownedCounter == 4] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 22: [LinkDownedCounter == 4] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 23: [LinkDownedCounter == 5] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 24: [LinkDownedCounter == 5] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 26: [LinkDownedCounter == 10] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 27: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 28: [LinkDownedCounter == 5] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 29: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 2] [PortXmitWait == 1]

         GUID 0x2c903008ccf00 port 30: [PortRcvSwitchRelayErrors == 2]

         GUID 0x2c903008ccf00 port 31: [LinkDownedCounter == 4] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 32: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 34: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 35: [LinkDownedCounter == 9] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008ccf00 port 36: [PortRcvRemotePhysicalErrors == 16231] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

      Errors for "node106 HCA-1"

         GUID 0xe8611f104a85 port 1: [PortXmitWait == 370282]

      Errors for "node105 HCA-1"

         GUID 0xe8611f104bc5 port 1: [PortXmitWait == 1415449]

      Errors for "node10 HCA-1"

         GUID 0x2c90300419071 port 1: [PortXmitWait == 1682321]

      Errors for "node104 HCA-1"

         GUID 0xe8611f104995 port 1: [PortXmitWait == 2769787]

      Errors for "node7 HCA-1"

         GUID 0x2c903001bb811 port 1: [PortXmitWait == 6544359]

      Errors for "node101 HCA-1"

         GUID 0xe8611f104b35 port 1: [PortXmitWait == 599737]

      Errors for "node6 HCA-1"

         GUID 0x2c903001ba801 port 1: [PortXmitWait == 56043]

      Errors for "node5 HCA-1"

         GUID 0x2c90300357551 port 1: [PortXmitWait == 52033813]

      Errors for "node3 HCA-1"

         GUID 0x2c903001ba4b1 port 1: [PortXmitWait == 25888335]

      Errors for "io4 HCA-1"

         GUID 0xf452140300338911 port 1: [PortXmitWait == 20]

      Errors for "io3 HCA-1"

         GUID 0xf452140300339751 port 1: [PortXmitWait == 427]

      Errors for "io2 HCA-1"

         GUID 0xf452140300339731 port 1: [PortXmitWait == 2658548]

      Errors for "io1 HCA-1"

         GUID 0xf452140300339471 port 1: [PortXmitWait == 263]

      Errors for "tc4600 HCA-1"

         GUID 0xf45214030082cab1 port 1: [PortXmitWait == 1735150]

      Errors for 0x2c903008d5600 "SwitchX -  Mellanox Technologies"

         GUID 0x2c903008d5600 port ALL: [LinkDownedCounter == 255] [PortRcvErrors == 2] [PortRcvRemotePhysicalErrors == 45963] [PortRcvSwitchRelayErrors == 27939] [PortXmitDiscards == 1927] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 1: [PortXmitWait == 98295522]

         GUID 0x2c903008d5600 port 2: [PortXmitWait == 51845652]

         GUID 0x2c903008d5600 port 3: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 4: [PortRcvSwitchRelayErrors == 3] [PortXmitDiscards == 1079] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 5: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 6: [LinkDownedCounter == 2] [PortRcvSwitchRelayErrors == 123] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 7: [PortRcvSwitchRelayErrors == 27159] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 8: [PortRcvSwitchRelayErrors == 13] [PortXmitDiscards == 161] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 9: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 116] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 10: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 11: [PortRcvRemotePhysicalErrors == 128] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 12: [PortRcvRemotePhysicalErrors == 38462] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 13: [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 14: [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 15: [PortRcvRemotePhysicalErrors == 2016] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 16: [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 17: [PortRcvRemotePhysicalErrors == 219] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 18: [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 19: [PortRcvRemotePhysicalErrors == 74] [PortXmitWait == 674164601]

         GUID 0x2c903008d5600 port 20: [PortRcvRemotePhysicalErrors == 4983] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 21: [PortRcvSwitchRelayErrors == 2] [PortXmitWait == 281438]

         GUID 0x2c903008d5600 port 22: [PortRcvRemotePhysicalErrors == 81] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 23: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 220212]

         GUID 0x2c903008d5600 port 24: [LinkDownedCounter == 66] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 25: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 2198114]

         GUID 0x2c903008d5600 port 26: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 27: [PortRcvErrors == 2] [PortRcvSwitchRelayErrors == 3] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 29: [LinkDownedCounter == 202] [PortRcvSwitchRelayErrors == 241] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 30: [LinkDownedCounter == 2] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 31: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4257910]

         GUID 0x2c903008d5600 port 33: [PortRcvSwitchRelayErrors == 381] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 34: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 323] [PortXmitWait == 4294967295]

         GUID 0x2c903008d5600 port 35: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 484796]

         GUID 0x2c903008d5600 port 36: [PortRcvSwitchRelayErrors == 2] [PortXmitDiscards == 248] [PortXmitWait == 4294967295]

      Errors for "node99 HCA-1"

         GUID 0xe8611f104a75 port 1: [PortXmitWait == 9345108]

      Errors for "node100 HCA-1"

         GUID 0xe8611f104a55 port 1: [PortXmitWait == 22436]

      Errors for "node97 HCA-1"

         GUID 0xe8611f104a95 port 1: [PortXmitWait == 3856160]

      Errors for "node98 HCA-1"

         GUID 0xe8611f104ae5 port 1: [PortXmitWait == 28418]

      Errors for "node96 HCA-1"

         GUID 0xe8611f104675 port 1: [PortXmitWait == 3173368]

      Errors for "node95 HCA-1"

         GUID 0xe8611f104665 port 1: [PortXmitWait == 4704227]

      Errors for "node94 HCA-1"

         GUID 0xe8611f104735 port 1: [PortXmitWait == 5456975]

      Errors for "node93 HCA-1"

         GUID 0xe8611f104725 port 1: [PortXmitWait == 2931296]

      Errors for "node92 HCA-1"

         GUID 0xe8611f1046c5 port 1: [PortXmitWait == 19874]

      Errors for "node91 HCA-1"

         GUID 0xe8611f104ab5 port 1: [PortXmitWait == 550147]

      Errors for "node72 HCA-1"

         GUID 0xe8611f104945 port 1: [PortXmitWait == 215794]

      Errors for "node71 HCA-1"

         GUID 0xe8611f104935 port 1: [PortXmitWait == 29770]

      Errors for "node70 HCA-1"

         GUID 0xe8611f1047a5 port 1: [PortXmitWait == 26135]

      Errors for "node69 HCA-1"

         GUID 0xe8611f1046e5 port 1: [PortXmitWait == 8863391]

      Errors for "node68 HCA-1"

         GUID 0xe8611f104975 port 1: [PortXmitWait == 1319]

      Errors for "node67 HCA-1"

         GUID 0xe8611f1045f5 port 1: [PortXmitWait == 3451879]

      Errors for "node66 HCA-1"

         GUID 0xe8611f1048d5 port 1: [PortXmitWait == 1255020]

      Errors for "node65 HCA-1"

         GUID 0xe8611f1048f5 port 1: [PortXmitWait == 98193]

      Errors for "node64 HCA-1"

         GUID 0xe8611f1048e5 port 1: [PortXmitWait == 2364]

      Errors for "node63 HCA-1"

         GUID 0xe8611f1047c5 port 1: [PortXmitWait == 37412]

      Errors for "node62 HCA-1"

         GUID 0xe8611f1049b5 port 1: [PortXmitWait == 8185167]

      Errors for "node61 HCA-1"

         GUID 0xe8611f104785 port 1: [PortXmitWait == 1464886]

      Errors for "node60 HCA-1"

         GUID 0xe8611f1048a5 port 1: [PortXmitWait == 2090245]

      Errors for "node59 HCA-1"

         GUID 0xe8611f104855 port 1: [PortXmitWait == 5425108]

      Errors for "node58 HCA-1"

         GUID 0xe8611f104895 port 1: [PortXmitWait == 10235]

      Errors for "node57 HCA-1"

         GUID 0xe8611f104625 port 1: [PortXmitWait == 11568]

      Errors for "node56 HCA-1"

         GUID 0xe8611f104ad5 port 1: [PortXmitWait == 10900]

      Errors for "node55 HCA-1"

         GUID 0xe8611f104ac5 port 1: [PortXmitWait == 8138]

      Errors for "node54 HCA-1"

         GUID 0xe8611f104b25 port 1: [PortXmitWait == 19559]

      Errors for "node53 HCA-1"

         GUID 0xe8611f104b15 port 1: [PortXmitWait == 3337]

      Errors for "node52 HCA-1"

         GUID 0xe8611f104a65 port 1: [PortXmitWait == 2305000]

      Errors for "node51 HCA-1"

         GUID 0xe8611f1048b5 port 1: [PortXmitWait == 2859967]

      Errors for "node50 HCA-1"

         GUID 0xe8611f104875 port 1: [PortXmitWait == 142054]

      Errors for "node47 HCA-1"

         GUID 0xe8611f1046b5 port 1: [PortXmitWait == 384681]

      Errors for "node46 HCA-1"

         GUID 0xe8611f102bf9 port 1: [PortXmitWait == 113900]

      Errors for "node45 HCA-1"

         GUID 0xe8611f102c49 port 1: [PortXmitWait == 1]

      Errors for "node44 HCA-1"

         GUID 0xe8611f102dd9 port 1: [PortXmitWait == 2109061]

      Errors for "node43 HCA-1"

         GUID 0xe8611f102be9 port 1: [PortXmitWait == 1492859]

      Errors for "node41 HCA-1"

         GUID 0xe8611f102c29 port 1: [PortXmitWait == 1319431]

      Errors for "node42 HCA-1"

         GUID 0xe8611f102db9 port 1: [PortXmitWait == 1767831]

      Errors for "node40 HCA-1"

         GUID 0xe8611f102c79 port 1: [PortXmitWait == 8022]

      Errors for "node39 HCA-1"

         GUID 0xe8611f102d39 port 1: [PortXmitWait == 1275902]

      Errors for "node38 HCA-1"

         GUID 0xe8611f102bb9 port 1: [PortXmitWait == 681445]

      Errors for "node37 HCA-1"

         GUID 0xe8611f102ca9 port 1: [PortXmitWait == 149394]

      Errors for "node36 HCA-1"

         GUID 0xe8611f102c99 port 1: [PortXmitWait == 2817]

      Errors for "node35 HCA-1"

         GUID 0xe8611f102b99 port 1: [PortXmitWait == 4314308]

      Errors for "node34 HCA-1"

         GUID 0xe8611f102e49 port 1: [PortXmitWait == 4357]

      Errors for "node33 HCA-1"

         GUID 0xe8611f102d69 port 1: [PortXmitWait == 2630012]

      Errors for "node32 HCA-1"

         GUID 0xe8611f102b79 port 1: [PortXmitWait == 6621266]

      Errors for "node31 HCA-1"

         GUID 0xe8611f102b39 port 1: [PortXmitWait == 2057]

      Errors for "node30 HCA-1"

         GUID 0xe8611f102b29 port 1: [PortXmitWait == 13528]

      Errors for "node29 HCA-1"

         GUID 0xe8611f102cd9 port 1: [PortXmitWait == 674]

      Errors for "node28 HCA-1"

         GUID 0xe8611f102da9 port 1: [PortXmitWait == 1597]

      Errors for "node27 HCA-1"

         GUID 0xe8611f102d09 port 1: [PortXmitWait == 189324]

      Errors for "node26 HCA-1"

         GUID 0xe8611f102c09 port 1: [PortXmitWait == 443]

      Errors for "node25 HCA-1"

         GUID 0xe8611f102d59 port 1: [PortXmitWait == 1691]

      Errors for "node24 HCA-1"

         GUID 0xe8611f102b89 port 1: [PortXmitWait == 18]

      Errors for "node23 HCA-1"

         GUID 0xe8611f102cb9 port 1: [PortXmitWait == 69911]

      Errors for "node22 HCA-1"

         GUID 0xe8611f102d49 port 1: [PortXmitWait == 23050]

      Errors for "node21 HCA-1"

         GUID 0xe8611f102d29 port 1: [PortXmitWait == 5232157]

      Errors for "node20 HCA-1"

         GUID 0xe8611f102bd9 port 1: [PortXmitWait == 200427]

      Errors for "node19 HCA-1"

         GUID 0xe8611f102b49 port 1: [PortXmitWait == 2110]

      Errors for "node18 HCA-1"

         GUID 0xe8611f102c19 port 1: [PortXmitWait == 5403]

      Errors for "node17 HCA-1"

         GUID 0xe8611f102ce9 port 1: [PortXmitWait == 261964]

      Errors for "node16 HCA-1"

         GUID 0xe8611f102b69 port 1: [PortXmitWait == 2790552]

      Errors for "node15 HCA-1"

         GUID 0xe8611f102ba9 port 1: [PortXmitWait == 5652]

      Errors for "node14 HCA-1"

         GUID 0xe8611f102c69 port 1: [PortXmitWait == 3789]

      Errors for "node13 HCA-1"

         GUID 0xe8611f102e59 port 1: [PortXmitWait == 2459]

      Errors for "node12 HCA-1"

         GUID 0xe8611f102b59 port 1: [PortXmitWait == 1023]

      Errors for "node11 HCA-1"

         GUID 0xe8611f102d19 port 1: [PortXmitWait == 136582]

      Errors for "node10 HCA-1"

         GUID 0xe8611f102e29 port 1: [PortXmitWait == 381880]

      Errors for "node9 HCA-1"

         GUID 0xe8611f102cf9 port 1: [PortXmitWait == 7573536]

      Errors for "node8 HCA-1"

         GUID 0xe8611f102e39 port 1: [PortXmitWait == 924132]

      Errors for "node7 HCA-1"

         GUID 0xe8611f102e19 port 1: [PortXmitWait == 21617]

      Errors for "node6 HCA-1"

         GUID 0xe8611f102e69 port 1: [PortXmitWait == 2949430]

      Errors for "node5 HCA-1"

         GUID 0xe8611f102c39 port 1: [PortXmitWait == 2317637]

      Errors for "node4 HCA-1"

         GUID 0xe8611f102d79 port 1: [PortXmitWait == 6303]

      Errors for "node3 HCA-1"

         GUID 0xe8611f102bc9 port 1: [PortXmitWait == 4568803]

      Errors for "node2 HCA-1"

         GUID 0xe8611f102c59 port 1: [PortXmitWait == 94382]

      Errors for "node1 HCA-1"

         GUID 0xe8611f102dc9 port 1: [PortXmitWait == 841]

      Errors for 0xf4521403008b76c0 "MF0;switch-993d34:SX6506/L06/U1"

         GUID 0xf4521403008b76c0 port ALL: [LinkDownedCounter == 11] [PortRcvSwitchRelayErrors == 22802] [PortXmitDiscards == 8057] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b76c0 port 1: [PortRcvSwitchRelayErrors == 3] [PortXmitDiscards == 1079] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b76c0 port 2: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 336] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b76c0 port 3: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 3678] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b76c0 port 4: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 129] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b76c0 port 5: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 522] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b76c0 port 6: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 198] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b76c0 port 7: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 1870] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b76c0 port 8: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 204] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b76c0 port 9: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 41] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b76c0 port 10: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b76c0 port 11: [LinkDownedCounter == 2] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 1771]

         GUID 0xf4521403008b76c0 port 12: [PortXmitWait == 571701]

         GUID 0xf4521403008b76c0 port 13: [LinkDownedCounter == 3] [PortRcvSwitchRelayErrors == 2] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b76c0 port 14: [PortXmitWait == 9250848]

         GUID 0xf4521403008b76c0 port 15: [LinkDownedCounter == 2] [PortRcvSwitchRelayErrors == 5817] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b76c0 port 16: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 6027] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b76c0 port 17: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 5569] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b76c0 port 18: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 5374] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b76c0 port 19: [PortXmitWait == 3297636]

         GUID 0xf4521403008b76c0 port 20: [PortXmitWait == 65313094]

         GUID 0xf4521403008b76c0 port 21: [PortXmitWait == 1165031]

         GUID 0xf4521403008b76c0 port 22: [PortXmitWait == 5774741]

         GUID 0xf4521403008b76c0 port 23: [PortXmitWait == 2620130]

         GUID 0xf4521403008b76c0 port 24: [PortXmitWait == 4708293]

         GUID 0xf4521403008b76c0 port 25: [PortXmitWait == 8904962]

         GUID 0xf4521403008b76c0 port 26: [PortXmitWait == 27577749]

         GUID 0xf4521403008b76c0 port 27: [PortXmitWait == 29928363]

         GUID 0xf4521403008b76c0 port 28: [PortXmitWait == 12385189]

         GUID 0xf4521403008b76c0 port 29: [PortXmitWait == 16828511]

         GUID 0xf4521403008b76c0 port 30: [PortXmitWait == 39950914]

         GUID 0xf4521403008b76c0 port 31: [PortXmitWait == 7779423]

         GUID 0xf4521403008b76c0 port 32: [PortXmitWait == 972493]

         GUID 0xf4521403008b76c0 port 33: [PortXmitWait == 24165400]

         GUID 0xf4521403008b76c0 port 34: [PortXmitWait == 9421146]

         GUID 0xf4521403008b76c0 port 35: [PortXmitWait == 4778723]

         GUID 0xf4521403008b76c0 port 36: [PortXmitWait == 1660990943]

      Errors for 0xf4521403008b6fa0 "MF0;switch-993d34:SX6506/L04/U1"

         GUID 0xf4521403008b6fa0 port ALL: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 270] [PortXmitDiscards == 105] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 1: [PortRcvSwitchRelayErrors == 3] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 2: [PortRcvSwitchRelayErrors == 2] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 3: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 4: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 5: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 6: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 7: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 8: [PortRcvSwitchRelayErrors == 74] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 9: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 10: [PortRcvSwitchRelayErrors == 8] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 11: [PortRcvSwitchRelayErrors == 2] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 12: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 13: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 14: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 15: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 16: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 17: [PortRcvSwitchRelayErrors == 169] [PortXmitDiscards == 79] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 18: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 20] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 19: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 20: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 21: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 22: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 23: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 24: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 25: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 26: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 27: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 28: [PortXmitDiscards == 6] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 29: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 30: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 31: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 32: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 33: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 34: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 35: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b6fa0 port 36: [PortXmitWait == 4294967295]

      Errors for 0xf4521403008b7060 "MF0;switch-993d34:SX6506/L03/U1"

         GUID 0xf4521403008b7060 port ALL: [LinkDownedCounter == 12] [PortRcvSwitchRelayErrors == 22] [PortXmitDiscards == 134] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 1: [PortRcvSwitchRelayErrors == 5] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 2: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 3: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 4: [LinkDownedCounter == 2] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 5: [LinkDownedCounter == 2] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 6: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 10747]

         GUID 0xf4521403008b7060 port 7: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 36858]

         GUID 0xf4521403008b7060 port 8: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 84156103]

         GUID 0xf4521403008b7060 port 9: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 10: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 11: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 12: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 13: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 14: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 15: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 16: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 17: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 18: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 19: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 20: [PortXmitDiscards == 59] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 21: [PortXmitDiscards == 68] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 22: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 23: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 24: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 25: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 26: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 27: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 28: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 29: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 30: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 31: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 32: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 33: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 34: [PortXmitDiscards == 7] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 35: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7060 port 36: [PortXmitWait == 4294967295]

      Errors for 0xf4521403008b68d0 "MF0;switch-993d34:SX6506/L02/U1"

         GUID 0xf4521403008b68d0 port ALL: [LinkDownedCounter == 255] [PortRcvSwitchRelayErrors == 19435] [PortXmitDiscards == 18511] [LocalLinkIntegrityErrors == 15] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 1: [PortRcvSwitchRelayErrors == 4] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 2: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 3: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 4: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 5: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 6: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 7: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 8: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 9: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 10: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 11: [PortRcvSwitchRelayErrors == 581] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 12: [PortRcvSwitchRelayErrors == 1444] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 13: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 14: [PortRcvSwitchRelayErrors == 17391] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 15: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 16: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 17: [LinkDownedCounter == 3] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 18: [LinkDownedCounter == 255] [LocalLinkIntegrityErrors == 15] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 19: [PortRcvSwitchRelayErrors == 2] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 20: [PortXmitDiscards == 4596] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 21: [PortXmitDiscards == 6650] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 22: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 23: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 24: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 25: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 26: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 27: [PortXmitDiscards == 1155] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 28: [PortXmitDiscards == 5403] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 29: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 30: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 31: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 32: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 33: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 34: [PortXmitDiscards == 707] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 35: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b68d0 port 36: [PortXmitWait == 4294967295]

      Errors for 0xf4521403008b7300 "MF0;switch-993d34:SX6506/L01/U1"

         GUID 0xf4521403008b7300 port ALL: [LinkDownedCounter == 9] [PortRcvSwitchRelayErrors == 349] [PortXmitDiscards == 2898] [VL15Dropped == 22] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 1: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 2: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 3: [PortRcvSwitchRelayErrors == 15] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 4: [PortRcvSwitchRelayErrors == 4] [VL15Dropped == 22] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 5: [LinkDownedCounter == 3] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 6: [PortRcvSwitchRelayErrors == 4] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 7: [PortRcvSwitchRelayErrors == 10] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 8: [PortRcvSwitchRelayErrors == 2] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 9: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 10: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 11: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 12: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 13: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 14: [PortRcvSwitchRelayErrors == 247] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 15: [LinkDownedCounter == 5] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 16: [PortRcvSwitchRelayErrors == 56] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 17: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 18: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 19: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 20: [PortXmitDiscards == 884] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 21: [PortXmitDiscards == 1118] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 22: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 23: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 24: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 25: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 26: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 27: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 28: [PortXmitDiscards == 648] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 29: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 30: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 31: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 32: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 33: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 34: [PortXmitDiscards == 248] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 35: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7300 port 36: [PortXmitWait == 4294967295]

      Errors for 0xf452140300476130 "MF0;switch-993d34:SX6506/S03/U1"

         GUID 0xf452140300476130 port ALL: [PortXmitDiscards == 291] [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 1: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 2: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 3: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 4: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 5: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 6: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 7: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 8: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 9: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 10: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 11: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 12: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 13: [PortXmitWait == 601]

         GUID 0xf452140300476130 port 14: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 15: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 16: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 17: [PortXmitWait == 3499372837]

         GUID 0xf452140300476130 port 18: [PortXmitWait == 2966702]

         GUID 0xf452140300476130 port 19: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 20: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 21: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 22: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 23: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 24: [PortXmitDiscards == 291] [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 25: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 26: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 27: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 28: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 29: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 30: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 34: [PortXmitWait == 4294967295]

         GUID 0xf452140300476130 port 35: [PortXmitWait == 2967821492]

         GUID 0xf452140300476130 port 36: [PortXmitWait == 4280804153]

      Errors for 0xf452140300476100 "MF0;switch-993d34:SX6506/S02/U1"

         GUID 0xf452140300476100 port ALL: [PortXmitDiscards == 343] [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 0: [PortXmitWait == 15675680]

         GUID 0xf452140300476100 port 1: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 2: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 3: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 4: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 5: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 6: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 7: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 8: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 9: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 10: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 11: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 12: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 13: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 14: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 15: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 16: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 17: [PortXmitDiscards == 323] [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 18: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 19: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 20: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 21: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 22: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 23: [PortXmitDiscards == 20] [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 24: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 25: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 26: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 27: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 28: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 29: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 30: [PortXmitWait == 4294967295]

         GUID 0xf452140300476100 port 31: [PortXmitWait == 498646609]

         GUID 0xf452140300476100 port 32: [PortXmitWait == 796440845]

         GUID 0xf452140300476100 port 33: [PortXmitWait == 293780019]

      Errors for "node89 HCA-1"

         GUID 0xe8611f104775 port 1: [PortXmitWait == 519674]

      Errors for 0xf452140300476080 "MF0;switch-993d34:SX6506/S01/U1"

         GUID 0xf452140300476080 port ALL: [PortXmitDiscards == 1603] [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 0: [PortXmitWait == 3773520]

         GUID 0xf452140300476080 port 1: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 2: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 3: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 4: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 5: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 6: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 7: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 8: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 9: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 10: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 11: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 12: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 13: [PortXmitWait == 44639]

         GUID 0xf452140300476080 port 14: [PortXmitWait == 174723]

         GUID 0xf452140300476080 port 15: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 16: [PortXmitWait == 33152813]

         GUID 0xf452140300476080 port 17: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 18: [PortXmitWait == 137950884]

         GUID 0xf452140300476080 port 19: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 20: [PortXmitDiscards == 2] [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 21: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 22: [PortXmitDiscards == 1601] [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 23: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 24: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 25: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 26: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 27: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 28: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 29: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 30: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 31: [PortXmitWait == 4294967295]

         GUID 0xf452140300476080 port 32: [PortXmitWait == 145892]

         GUID 0xf452140300476080 port 33: [PortXmitWait == 5768275]

         GUID 0xf452140300476080 port 34: [PortXmitWait == 15651998]

         GUID 0xf452140300476080 port 35: [PortXmitWait == 301315601]

         GUID 0xf452140300476080 port 36: [PortXmitWait == 2095531879]

      Errors for "node88 HCA-1"

         GUID 0xe8611f1046f5 port 1: [PortXmitWait == 36631]

      Errors for "node87 HCA-1"

         GUID 0xe8611f104755 port 1: [PortXmitWait == 178718]

      Errors for "node85 HCA-1"

         GUID 0xe8611f1046d5 port 1: [PortXmitWait == 692609]

      Errors for "node86 HCA-1"

         GUID 0xe8611f104645 port 1: [PortXmitWait == 205382]

      Errors for "node83 HCA-1"

         GUID 0xe8611f104695 port 1: [PortXmitWait == 1159110]

      Errors for "node82 HCA-1"

         GUID 0xe8611f104715 port 1: [PortXmitWait == 531365]

      Errors for "node80 HCA-1"

         GUID 0xe8611f104765 port 1: [PortXmitWait == 44858]

      Errors for "node81 HCA-1"

         GUID 0xe8611f1049d5 port 1: [PortXmitWait == 337661]

      Errors for "node79 HCA-1"

         GUID 0xe8611f104915 port 1: [PortXmitWait == 26707]

      Errors for "node78 HCA-1"

         GUID 0xe8611f104965 port 1: [PortXmitWait == 79891]

      Errors for "node77 HCA-1"

         GUID 0xe8611f1046a5 port 1: [PortXmitWait == 1177117]

      Errors for "node76 HCA-1"

         GUID 0xe8611f104a45 port 1: [PortXmitWait == 2876827]

      Errors for "node74 HCA-1"

         GUID 0xe8611f104605 port 1: [PortXmitWait == 39148]

      Errors for "node73 HCA-1"

         GUID 0xe8611f1049c5 port 1: [PortXmitWait == 2293292]

      Errors for 0xf4521403008b7720 "MF0;switch-993d34:SX6506/L05/U1"

         GUID 0xf4521403008b7720 port ALL: [LinkDownedCounter == 8] [PortRcvSwitchRelayErrors == 176] [PortXmitDiscards == 784] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 1: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 102] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 2: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 3: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 4: [LinkDownedCounter == 6] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 5: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 6: [PortRcvSwitchRelayErrors == 3] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 7: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 8: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 39] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 9: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 10: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 46] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 11: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 12: [PortRcvSwitchRelayErrors == 1] [PortXmitDiscards == 383] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 13: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 14: [PortRcvSwitchRelayErrors == 157] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 15: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 16: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 17: [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 18: [LinkDownedCounter == 1] [PortRcvSwitchRelayErrors == 1] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 19: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 20: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 21: [PortXmitWait == 372817446]

         GUID 0xf4521403008b7720 port 22: [PortXmitWait == 5065172]

         GUID 0xf4521403008b7720 port 23: [PortXmitWait == 42903661]

         GUID 0xf4521403008b7720 port 24: [PortXmitWait == 2743754029]

         GUID 0xf4521403008b7720 port 25: [PortXmitWait == 978145227]

         GUID 0xf4521403008b7720 port 26: [PortXmitWait == 1575017077]

         GUID 0xf4521403008b7720 port 27: [PortXmitDiscards == 214] [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 28: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 29: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 34: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 35: [PortXmitWait == 4294967295]

         GUID 0xf4521403008b7720 port 36: [PortXmitWait == 4294967295]

      Errors for "node84 HCA-1"

         GUID 0xe8611f104685 port 1: [PortXmitWait == 8651819]

       

      ## Summary: 175 nodes checked, 167 bad nodes found

      ##          597 ports checked, 552 ports have errors beyond threshold

      ## Thresholds:

      ## Suppressed:

      Can you help me? Thank you very much.