0 Replies Latest reply on Sep 3, 2015 2:02 PM by m.gh

    Cannot achieve 100Gbps with MCX416A-CCAT?

    m.gh

      Hello!

       

      We have 2 Mellanox MCX416A-CCAT(100G Ethernet NIC Card) installed on two Dell Server R720xd. We have used iperf software to generate traffic from server A to server B (6 parallel thread). The maximum achievable bandwidth is near 64Gb/s. My question is why we cannot achieve 100Gb/s bandwidth?

       

      We have the latest driver version and the NICs have been optimised to use multiple cpus.

       

      Here is our complementary information for debugging

       

      Thanks!

       

      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

      Cable:

      • TE QSFP28 26AWG 3m

      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

      Server:

      • 2×INTEL XEON 8CORE E5-2690 2.90GHZ 20MB L3 CACHE PROCESSORS
      • 128 GB (4×32) MEMORY
      • PCIE3x16lanes

      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

      NIC:

      • 2x MCX416A-CCAT

      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

      OS:

      Linux 3.11.0-22-generic #38~precise1-Ubuntu SMP Fri May 16 20:47:57 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

      *-network:0

             description: Ethernet interface

             product: MT27620 Family

             vendor: Mellanox Technologies

             physical id: 0

             bus info: pci@0000:05:00.0

             logical name: eth12

             version: 00

             serial: e4:1d:2d:e9:81:9e

             width: 64 bits

             clock: 33MHz

             capabilities: bus_master cap_list ethernet physical fibre

             configuration: autonegotiation=off broadcast=yes driver=mlx5_core driverversion=3.0-1.0.1 (03 Mar 2015) duplex=full firmware=12.100.6440 ip=192.168.12.112 latency=0 link=yes multicast=yes

             resources: irq:48 memory:d0000000-d1ffffff

       

      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

      Settings for eth12:

        Supported ports: [ FIBRE ]

        Supported link modes:   Not reported

        Supported pause frame use: Symmetric Receive-only

        Supports auto-negotiation: No

        Advertised link modes:  Not reported

        Advertised pause frame use: No

        Advertised auto-negotiation: No

        Speed: 100000Mb/s

        Duplex: Full

        Port: Direct Attach Copper

        PHYAD: 0

        Transceiver: internal

        Auto-negotiation: off

      Cannot get wake-on-lan settings: Operation not permitted

        Link detected: yes

      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

      ethtool -i eth12:

       

      driver: mlx5_core

      version: 3.0-1.0.1 (03 Mar 2015)

      firmware-version: 12.100.6440

      bus-info: 0000:05:00.0

      supports-statistics: yes

      supports-test: no

      supports-eeprom-access: no

      supports-register-dump: no

      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

      ifconfig:

       

      eth12     Link encap:Ethernet  HWaddr ...

                inet addr:192.168.12.112  Bcast:192.168.12.255  Mask:255.255.255.0

                inet6 addr: fe80::e61d:2dff:fee9:819e/64 Scope:Link

                UP BROADCAST RUNNING MULTICAST  MTU:9000  Metric:1

                RX packets:38207802 errors:0 dropped:0 overruns:0 frame:0

                TX packets:427379787 errors:0 dropped:0 overruns:0 carrier:0

                collisions:0 txqueuelen:1000

                RX bytes:2292477406 (2.2 GB)  TX bytes:3849436789889 (3.8 TB)

      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

      05:00.0 Network controller [0207]: Mellanox Technologies MT27620 Family

        Subsystem: Mellanox Technologies Device 0008

        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+

        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

        Latency: 0, Cache Line Size: 64 bytes

        Interrupt: pin A routed to IRQ 48

        Region 0: Memory at d0000000 (64-bit, prefetchable) [size=32M]

        Capabilities: <access denied>

        Kernel driver in use: mlx5_core

        Kernel modules: mlx5_core

      ************************************************************

        Designation: PCI6

        Type: x16 PCI Express 3

       

      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

      Discovered irqs for eth12: 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222

      Assign irq 191 core_id 15

      Assign irq 192 core_id 16

      Assign irq 193 core_id 17

      Assign irq 194 core_id 18

      Assign irq 195 core_id 15

      Assign irq 196 core_id 16

      Assign irq 197 core_id 17

      Assign irq 198 core_id 18

      Assign irq 199 core_id 15

      Assign irq 200 core_id 16

      Assign irq 201 core_id 17

      Assign irq 202 core_id 18

      Assign irq 203 core_id 15

      Assign irq 204 core_id 16

      Assign irq 205 core_id 17

      Assign irq 206 core_id 18

      Assign irq 207 core_id 15

      Assign irq 208 core_id 16

      Assign irq 209 core_id 17

      Assign irq 210 core_id 18

      Assign irq 211 core_id 15

      Assign irq 212 core_id 16

      Assign irq 213 core_id 17

      Assign irq 214 core_id 18

      Assign irq 215 core_id 15

      Assign irq 216 core_id 16

      Assign irq 217 core_id 17

      Assign irq 218 core_id 18

      Assign irq 219 core_id 15

      Assign irq 220 core_id 16

      Assign irq 221 core_id 17

      Assign irq 222 core_id 18

      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

      cpuinfo: 32 cpu

       

      processor : 31

      vendor_id : GenuineIntel

      cpu family : 6

      model : 45

      model name : Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz

      stepping : 7

      microcode : 0x710

      cpu MHz : 2900.109

      cache size : 20480 KB

      physical id : 1

      siblings : 16

      core id : 7

      cpu cores : 8

      apicid : 47

      initial apicid : 47

      fpu : yes

      fpu_exception : yes

      cpuid level : 13

      wp : yes

      flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm ida arat xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid

      bogomips : 5801.33

      clflush size : 64

      cache_alignment : 64

      address sizes : 46 bits physical, 48 bits virtual

      power management:

      +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

       

       

       

      Thanks!