3 Replies Latest reply on Feb 13, 2018 9:18 PM by blairo

    Unable to capture outgoing packets with ibdump

      Hi all,

       

      Appreciate any help with this. I am trying to capture network packets flowing to and from a machine with ConnectX-3 10GbE card using the ibdump utility. Somehow I am only able to see incoming packets in the captured file and not any outgoing packets at all.

       

      I can reproduce the issue by running sockperf ping-pong on the system and capturing with ibdump . (I have tried using sockperf with and without the --load-vma option which doesn't change anything). All I see are incoming packets.

       

      I use the VMA library so I have to install and use Mellanox OFED, so ibdump remains the only way I can capture packets in this setup.

       

      Useful info:

       

      OS: RHEL 6.4

      NIC: ConnectX-3 EN 10GbE

      Captured using: ibdump (no options since there is only one device and only one port is up)

      Analysed using: Wireshark

       

      $ ibdump -v:

      ibdump, 2.0.0-5, built on Aug 21 2013, 13:09:58. GIT Version: 41a4dd2

       

      $ ofed_info

      MLNX_OFED_LINUX-2.0-3.0.0 (OFED-2.0-3.0.0)

       

      $ ethtool -i eth2:

      driver: mlx4_en

      version: 2.1.6 (Aug 27 2013)

      firmware-version: 2.30.3110

      bus-info: 0000:05:00.0

      supports-statistics: yes

      supports-test: yes

      supports-eeprom-access: no

      supports-register-dump: no

      supports-priv-flags: no

       

       

      Any inputs are appreciated.

       

      Thanks,
      Sid

      Request to mods: Please post this question in the right forum if not already so. Thanks.

        • Re: Unable to capture outgoing packets with ibdump
          ferbs

          Hi,

           

          To use ibdump on ROCE, flow steering must be enabled :

           

          in /etc/modprobe.d/mlnx.conf :

          mlx4_core log_num_mgm_entry_size=-1


          restart driver


          After you set that I recommend you upgrade your driver to the latest 2.3-2. if that doesn't help, please try the version I attached.


          I think tcpdump is also a good option, depends on what you're trying to filter.



            • Re: Unable to capture outgoing packets with ibdump
              blairo

              Erez,

               

              Is it possible to use ibdump inside a VM with a ConnectX-3 SR-IOV VF based device passed through? Doesn't seem to be working for us:

               

              [root@hc13 ~]# ethtool -i mlx0

              driver: mlx4_en

              version: 3.4-1.0.0 (25 Sep 2016)

              firmware-version: 2.40.7000

              bus-info: 0000:00:06.0

              supports-statistics: yes

              supports-test: yes

              supports-eeprom-access: no

              supports-register-dump: no

              supports-priv-flags: yes

               

              [root@hc13 ~]# ibv_devinfo

              hca_id: mlx4_0

                      transport:                      InfiniBand (0)

                      fw_ver:                         2.40.7000

                      node_guid:                      0014:0500:8bcf:a171

                      sys_image_guid:                 f452:1403:0094:7c93

                      vendor_id:                      0x02c9

                      vendor_part_id:                 4100

                      hw_ver:                         0x1

                      board_id:                       MT_1100120019

                      phys_port_cnt:                  1

                      Device ports:

                              port:   1

                                      state:                  PORT_ACTIVE (4)

                                      max_mtu:                4096 (5)

                                      active_mtu:             4096 (5)

                                      sm_lid:                 0

                                      port_lid:               0

                                      port_lmc:               0x00

                                      link_layer:             Ethernet

               

              [root@hc13 ~]# show_gids

              DEV     PORT    INDEX   GID                                     IPv4            VER     DEV

              ---     ----    -----   ---                                     ------------    ---     ---

              mlx4_0  1       0       fe80:0000:0000:0000:3c7a:e8ff:fe84:ac60                 V1      mlx0

              mlx4_0  1       1       fdfd:eb1a:eb10:e400:3c7a:e8ff:fe84:ac60                 V1      mlx0

              mlx4_0  1       2       0000:0000:0000:0000:0000:ffff:ac10:e52f 172.16.229.47   V1      mlx0

               

              [root@hc13 ~]# ibdump -d mlx0

              Initiating resources ...

              searching for IB devices in host

              -E- IB device mlx0 wasn't found

              -E- failed to create resources

               

              [root@hc13 ~]# ibdump -d mlx4_0

              Initiating resources ...

              searching for IB devices in host

              -E- Failed to open mst device mlx4_0: Operation not supported

              -E- failed to create resources

            • Re: Unable to capture outgoing packets with ibdump
              blairo

              The modprobe line in the answer is missing the "options" keyword. Should be -

              /etc/modprobe.d/mlnx.conf:

              options mlx4_core log_num_mgm_entry_size=-1