7 Replies Latest reply on Feb 25, 2018 10:26 PM by jbravo

    Freebsd 11.1 ConnectX-4 VPI

    boliniak

      Hello I just have strange behaviour I'm using modules from Freebsd 11.1 I compiled and build using buildworld etc..  and I can load mlx5 without no issue but I don't see Infiniband port on system on dmesg I have

      mlx5_core0: <mlx5_core> mem 0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci1

      mlx5_core0: INFO: firmware version: 12.21.1000

      mlx5_core0: INFO: Module 0, status: plugged

      mlx5_core1: <mlx5_core> mem 0xf2000000-0xf3ffffff irq 17 at device 0.1 on pci1

      mlx5_core1: INFO: firmware version: 12.21.1000

       

      I have Infiniband switch and ports are in Infiniband mode , I have loaded mlx5  module (core module)  but why I can't see the ports ??

      Building this drivers from OFED 3.0 from Mellanox site end with :

      ............../drivex/mlx5/generated/freebsd/core/mlx5_alloc.c:28:10: fatal error: 'linux/errno.h' file not found

      #include <linux/errno.h>

               ^~~~~~~~~~~~~~~

      1 error generated.

       

      1   27 0xffffffff80200000 1f67a88  kernel

      2    1 0xffffffff82169000 316708   zfs.ko

      3    2 0xffffffff82480000 cb78     opensolaris.ko

      4    1 0xffffffff8248d000 3d908    mlx5.ko

      5    3 0xffffffff824cb000 1c960    linuxkpi.ko

      6    1 0xffffffff824e8000 b1770    ibcore.ko

      7    1 0xffffffff82621000 3650     ums.ko

      8    1 0xffffffff82625000 3c93f    linux64.ko

      9    1 0xffffffff82662000 7b0f     linux_common.ko

       

      Maybe someone can help with this ? I don't know why there is not mlx5_ib module ?? It is not implemented yet ??  Maybe Infiniband for ConnectX-4 VPI cards is not ready ??

      Please let me know.

       

      BR

      Adam

        • Re: Freebsd 11.1 ConnectX-4 VPI
          justinclift

          Hey Adam, as a thought... if you don't get a useful response here it's a good idea to ask on the FreeBSD Infiniband mailing list:

           

          freebsd-infiniband Info Page

           

          It's not a high traffic mailing list, but generally the people there are responsive to on-topic questions (like this).

            • Re: Freebsd 11.1 ConnectX-4 VPI
              march

              Hi Adam,

               

              You can try to build the inbox drivers

               

              cd /usr/src/sys/modules/mlx5

              make all install

              cd /usr/src/sys/modules/mlx5en

              make all install

              kldload mlx5

              ifconfig (must show your adapter)

               

              BR

              Marc

                • Re: Freebsd 11.1 ConnectX-4 VPI
                  jbravo

                  Hi Marc

                  Does this solve the directory reference problems like "linux/errno.h" ? Because if you pass this problem others come in the line. I think mellanox driver instructions on github page is old.Header file directory references must be organized in order to compile mellanox driver properly.Besides RDMA-ISER is not supported except initiator(there is another mellanox github page). Why target is not supported but initiator is something I wonder.After installation, freebsd-11-1 lists mellanox NIC properly.I dont get what the use of those drivers if rdma-iser is not supported

                  thnx

                    • Re: Freebsd 11.1 ConnectX-4 VPI
                      march

                      Hi,

                       

                      No, it doesn't solve the build error.

                      The build error is related to the mellanox driver sources that you got from git.

                      What i explained compiles the native driver provided with FreeBSD 11.1

                      For now, there's no GA release for FreeBSD 11.1 it is certainly the reason the build failed.

                      I 've tried to workaround this problem by adding in the CFLAGS the path to the linx/errno.h CFLAGs = -I<path>, but the compilation failed at a later stage.

                      Conclusion:

                      For now, there's only this FReeBSD 11 build that is supported : FreeBSD 11 Current (v289420)

                       

                      Marc

                        • Re: Freebsd 11.1 ConnectX-4 VPI
                          jbravo

                          Hi

                          So I should skip mellanox driver sources on github.What do you mean by inbox drivers?  Freebsd11.1 supports rdma-iser for both target and initiator?

                          thnx

                            • Re: Freebsd 11.1 ConnectX-4 VPI
                              justinclift

                              > What do you mean by inbox drivers?

                               

                              "Inbox drivers" is a weird term that some vendors (mostly Mellanox it seems) use to describe the drivers provided by the OS.

                               

                              My suspicion is that it's a shortened way to say "the drivers that came in the box".  I've not personally heard anyone other than Mellanox use the term until recently, where it was used by (from memory) a Red Hat engineer.  Hopefully the phrase dies out though, and people use the industry standard phrases instead.  eg "OS provided driver"

                                • Re: Freebsd 11.1 ConnectX-4 VPI
                                  jbravo

                                  "inbox driver" is a term that Mellanox uses for the drivers that comes with ubuntu centos debian rhel etc linux distributions.it is generic for all types of linux distros.this is the way of doing their business.

                                  My problem is that even if I dont install anything , freebsd11 lists connectx-4 VPI drivers.My question is that what the use of installing mellanox drivers on github is if freebsd inbox drivers are already there.

                                  Furthermore if ISER-RDMA is supported only for initiator but not target, I just concluded this because I did not get any response on this issue.initiator without target is like coffee without cookie.