1 Reply Latest reply on Jun 26, 2017 11:28 AM by alkx

    How to support MLX4

    chyan413

      Hi

      I am building VPP with Mellanox ConnectX-3 Pro NICs, I did a lot of attempts, but when I start vpp, there is still something wrong

      I did this with VPP master version,and mlx4 debug mode is on

       

      I did the following steps:

      1. dpdk/Makefile

         Add 2 lines similar to MLX5, and then complie dpdk with make "dpdk-install-dev DPDK_MLX4_PMD=y"

      2 search for MLX5 in following files and create simlar code for mlx4. Files are:

      src/plugins/dpdk/device/dpdk.h

      src/plugins/dpdk/device/format.c

      src/plugins/dpdk/device/init.c

      3 Build VPP+dpdk :

      # cd /vpp

      # make dpdk-install-dev DPDK_MLX4_PMD=y

      # make install-dep

      # make build

      # make pkg-deb vpp_uses_dpdk_mlx5_pmd=yes

      # dpkg -i build-root/vpp-lib*

      # dpkg -i build-root/vpp_17*

      # dpkg -i build-root/vpp-plugins*

       

      My NICS:

      root@ubuntu141:~/vSG/vpp# lspci | grep Eth

      03:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

      04:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

      05:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

      81:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

      84:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

       

      /etc/vpp/startup.conf :

      dpdk {

               dev 0000:03:00.0

               {

                  num-rx-queues 1

                  num-tx-queues 1

               }

      }

       

       

      from the following log,it seems some error in function from DPDK: mlx4_rx_queue_setup()-->rxq_setup()

      but I use standard DPDK, it recgnize the NICs OK and run pmdtest also OK

       

      vpp start log :

      EAL: Detected 48 lcore(s)

      EAL: No free hugepages reported in hugepages-1048576kB

      EAL: Probing VFIO support...

      EAL: PCI device 0000:03:00.0 on NUMA socket 0

      EAL:   probe driver: 15b3:1007 net_mlx4

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5549: mlx4_pci_probe(): using driver device index 0

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5571: mlx4_pci_probe(): checking device "mlx4_0"

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5582: mlx4_pci_probe(): PCI information matches, using device "mlx4_0" (VF: false)

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5602: mlx4_pci_probe(): device opened

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5605: mlx4_pci_probe(): 2 port(s) detected

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5627: mlx4_pci_probe(): using port 1 (00000001)

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5649: mlx4_pci_probe(): port 1 is not active: "down" (1)

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5702: mlx4_pci_probe(): device flags: IBV_DEVICE_QPG IBV_DEVICE_RSS

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5705: mlx4_pci_probe(): maximum RSS indirection table size: 256

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5714: mlx4_pci_probe(): checksum offloading is supported

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5719: mlx4_pci_probe(): L2 tunnel checksum offloads are supported

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5760: mlx4_pci_probe(): port 1 MAC address is 7c:fe:90:b2:26:f0

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5774: mlx4_pci_probe(): port 1 ifname is "p1p1"

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5781: mlx4_pci_probe(): port 1 MTU is 1500

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5837: mlx4_pci_probe(): forcing Ethernet interface up

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5627: mlx4_pci_probe(): using port 2 (00000002)

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5649: mlx4_pci_probe(): port 2 is not active: "down" (1)

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5702: mlx4_pci_probe(): device flags: IBV_DEVICE_QPG IBV_DEVICE_RSS

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5705: mlx4_pci_probe(): maximum RSS indirection table size: 256

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5714: mlx4_pci_probe(): checksum offloading is supported

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5719: mlx4_pci_probe(): L2 tunnel checksum offloads are supported

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5760: mlx4_pci_probe(): port 2 MAC address is 7c:fe:90:b2:26:f1

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5774: mlx4_pci_probe(): port 2 ifname is "p1p2"

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5781: mlx4_pci_probe(): port 2 MTU is 1500

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:5837: mlx4_pci_probe(): forcing Ethernet interface up

      DPDK physical memory layout:

      Segment 0: phys:0x35400000, len:4194304, virt:0x7fe512e00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 1: phys:0x38200000, len:2097152, virt:0x7fe512a00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 2: phys:0x38800000, len:2097152, virt:0x7fe512600000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 3: phys:0x39a00000, len:2097152, virt:0x7fe512200000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 4: phys:0x42400000, len:2097152, virt:0x7fe511e00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 5: phys:0x42800000, len:2097152, virt:0x7fe511a00000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 6: phys:0x42c00000, len:2097152, virt:0x7fe511600000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 7: phys:0x43000000, len:236978176, virt:0x7fe503200000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 8: phys:0x52600000, len:283115520, virt:0x7fe4f2200000, socket_id:0, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 9: phys:0x1e93e00000, len:2097152, virt:0x7fe489c00000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 10: phys:0x1e94200000, len:2097152, virt:0x7fe489800000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 11: phys:0x1e95e00000, len:2097152, virt:0x7fe489400000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 12: phys:0x1e97800000, len:2097152, virt:0x7fe489000000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 13: phys:0x1e98800000, len:2097152, virt:0x7fe488c00000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 14: phys:0x1e99e00000, len:2097152, virt:0x7fe488800000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 15: phys:0x1e9a400000, len:2097152, virt:0x7fe488400000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 16: phys:0x1e9a800000, len:2097152, virt:0x7fe488000000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 17: phys:0x1e9c800000, len:2097152, virt:0x7fe487c00000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 18: phys:0x1e9e800000, len:2097152, virt:0x7fe487800000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 19: phys:0x1e9ec00000, len:2097152, virt:0x7fe487400000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 20: phys:0x1ea2000000, len:2097152, virt:0x7fe487000000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 21: phys:0x1ea4e00000, len:2097152, virt:0x7fe486c00000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 22: phys:0x1ea6c00000, len:2097152, virt:0x7fe486800000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 23: phys:0x1ea7200000, len:2097152, virt:0x7fe486400000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 24: phys:0x1eab400000, len:2097152, virt:0x7fe486000000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 25: phys:0x1eac000000, len:2097152, virt:0x7fe485c00000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 26: phys:0x1eadc00000, len:2097152, virt:0x7fe485800000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 27: phys:0x1eae600000, len:2097152, virt:0x7fe485400000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 28: phys:0x1eb0e00000, len:2097152, virt:0x7fe485000000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 29: phys:0x1eb4e00000, len:2097152, virt:0x7fe484c00000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 30: phys:0x1eb6a00000, len:2097152, virt:0x7fe484800000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 31: phys:0x1eba600000, len:2097152, virt:0x7fe484400000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 32: phys:0x1ec2000000, len:2097152, virt:0x7fe484000000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 33: phys:0x1ec9600000, len:2097152, virt:0x7fe483c00000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 34: phys:0x1ecd000000, len:2097152, virt:0x7fe483800000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 35: phys:0x1ef5400000, len:2097152, virt:0x7fe483400000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 36: phys:0x1eff400000, len:2097152, virt:0x7fe483000000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 37: phys:0x1f5a400000, len:2097152, virt:0x7fe482c00000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 38: phys:0x1f5cc00000, len:2097152, virt:0x7fe482800000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      Segment 39: phys:0x1f5d000000, len:473956352, virt:0x7fe406600000, socket_id:1, hugepage_sz:2097152, nchannel:0, nrank:0

      0: dpdk_ipsec_process:241: DPDK Cryptodev support is disabled, default to OpenSSL IPsec

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:738: dev_configure(): 0x7fe560318d80: TX queues number update: 0 -> 1

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:753: dev_configure(): 0x7fe560318d80: RX queues number update: 0 -> 1

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:2046: mlx4_tx_queue_setup(): 0x7fe560318d80: configuring queue 0 for 1024 descriptors

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:1896: txq_setup(): priv->device_attr.max_qp_wr is 16351

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:1898: txq_setup(): priv->device_attr.max_sge is 32

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:1019: txq_alloc_elts(): 0x7fe56092b910: allocated and configured 256 WRs

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:2003: txq_setup(): 0x7fe512ebb8c0: cleaning-up old txq just in case

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:1109: txq_cleanup(): cleaning up 0x7fe512ebb8c0

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:1063: txq_free_elts(): 0x7fe512ebb8c0: freeing WRs

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:2006: txq_setup(): 0x7fe512ebb8c0: txq updated with 0x7fe56092b910

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:1372: txq_mp2mr(): 0x7fe512ebb8c0: discovered new memory pool "mbuf_pool_socket0" (0x7fe512efbec0)

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:1301: mlx4_mp2mr(): mempool 0x7fe512efbec0 area start=0x7fe50ecfffc0 end=0x7fe5113fffc0 size=40894464

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:1315: mlx4_mp2mr(): mempool 0x7fe512efbec0 using start=0x7fe50ec00000 end=0x7fe511400000 size=41943040 for MR

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:1393: txq_mp2mr(): 0x7fe512ebb8c0: new MR lkey for MP "mbuf_pool_socket0" (0x7fe512efbec0): 0x536938497

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:1372: txq_mp2mr(): 0x7fe512ebb8c0: discovered new memory pool "mbuf_pool_socket1" (0x7fe4226fbec0)

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:1301: mlx4_mp2mr(): mempool 0x7fe4226fbec0 area start=0x7fe41fffbe80 end=0x7fe4226fbe80 size=40894464

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:1315: mlx4_mp2mr(): mempool 0x7fe4226fbec0 using start=0x7fe41fe00000 end=0x7fe422800000 size=44040192 for MR

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:1393: txq_mp2mr(): 0x7fe512ebb8c0: new MR lkey for MP "mbuf_pool_socket1" (0x7fe4226fbec0): 0x536938753

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:2077: mlx4_tx_queue_setup(): 0x7fe560318d80: adding TX queue 0x7fe512ebb8c0 to list

      /home/stack/vSG/vpp/dpdk/deb/_build/dpdk-17.02/drivers/net/mlx4/mlx4.c:4018: mlx4_rx_queue_setup(): 0x7fe560318d80: configuring queue 0 for 1024 descriptors

      Segmentation fault (core dumped)