Mellanox Linux Driver Modules Relationship (MLNX_OFED)

Version 13

    This post describes the various modules of MLNX_OFED relations with the other Linux Kernel modules.

     

    Important notes:

    1. mlx5_ib, mlx5_core are used by Mellanox Connect-IB adapter cards, while mlx4_core, mlx4_en and mlx4_ib are used by ConnectX-3/ConnectX-3 Pro.
    2. ib_iser module is used by iscsi initiator, while ib_isert module is used by LIO iscsi target. TGT, being a userspace implementation, does not need any of those kernel modules.
      • ib_isert module is included in MLNX_OFED 2.4 and onward.
    3. As a troubleshooting action, you may need to check at first that the relevant modules are loaded (e.g. ib_iser, in case ISER initiator is being used, ib_isert in case LIO target is being used).
    4. Not all modules need to be loaded, it depends on the application and usage (e.g. no need to load mlx4_ib in case the link layer is Ethernet or vice versa, there is no need to load mlx4_en in case the link layer is InfiniBand).

     

    Related Linux commands:

    • lsmod:  provides a list of the kernel modules currently loaded on Linux O/S.
    • modprobe:  a command used to add or remove kernel modules on Linux O/S.
    • modinfo:   a command that shows information about the kernel module on Linux O/S.

     

    Module parameters:

    To check the value of a module parameter, run the following command

     

    cat /sys/module/<module-name>/parameters/<parameter-name>

     

    For example:

    # cat /sys/module/mlx4_core/parameters/roce_mode

    2

    The following diagram is high-level relationship between the relevant MLNX_OFED modules and the other kernel modules that uses them.

    Minor changes may apply depending on the various distribution and user applications.

     

    1%3Fauth_token%3D92c650be789233e1cb53156ab6136dac63ac90ec

     

    To find the full list of kernel modules replaced by the MLNX_OFED, run the following commands (or similar) that in this case it was taken in RHEL7 OS:

    # pwd

    /lib/modules/3.10.0-123.el7.x86_64/extra

     

    # find . * | grep .ko

    ./knem-mlnx/knem.ko

    ./mlnx-ofa_kernel/fs/9p/9p.ko

    ./mlnx-ofa_kernel/net/sunrpc/xprtrdma/svcrdma.ko

    ./mlnx-ofa_kernel/net/sunrpc/xprtrdma/xprtrdma.ko

    ./mlnx-ofa_kernel/net/rds/rds_tcp.ko

    ./mlnx-ofa_kernel/net/rds/rds.ko

    ./mlnx-ofa_kernel/net/rds/rds_rdma.ko

    ./mlnx-ofa_kernel/net/9p/9pnet_rdma.ko

    ./mlnx-ofa_kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko

    ./mlnx-ofa_kernel/drivers/infiniband/hw/nes/iw_nes.ko

    ./mlnx-ofa_kernel/drivers/infiniband/hw/mthca/ib_mthca.ko

    ./mlnx-ofa_kernel/drivers/infiniband/hw/qib/ib_qib.ko

    ./mlnx-ofa_kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko

    ./mlnx-ofa_kernel/drivers/infiniband/hw/ipath/ib_ipath.ko

    ./mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko

    ./mlnx-ofa_kernel/drivers/infiniband/hw/amso1100/iw_c2.ko

    ./mlnx-ofa_kernel/drivers/infiniband/hw/cxgb3/iw_cxgb3.ko

    ./mlnx-ofa_kernel/drivers/infiniband/ulp/isert/ib_isert.ko

    ./mlnx-ofa_kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko

    ./mlnx-ofa_kernel/drivers/infiniband/ulp/iser/ib_iser.ko

    ./mlnx-ofa_kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko

    ./mlnx-ofa_kernel/drivers/infiniband/ulp/srp/ib_srp.ko

    ./mlnx-ofa_kernel/drivers/infiniband/core/rdma_ucm.ko

    ./mlnx-ofa_kernel/drivers/infiniband/core/ib_mad.ko

    ./mlnx-ofa_kernel/drivers/infiniband/core/ib_core.ko

    ./mlnx-ofa_kernel/drivers/infiniband/core/ib_cm.ko

    ./mlnx-ofa_kernel/drivers/infiniband/core/ib_addr.ko

    ./mlnx-ofa_kernel/drivers/infiniband/core/rdma_cm.ko

    ./mlnx-ofa_kernel/drivers/infiniband/core/ib_umad.ko

    ./mlnx-ofa_kernel/drivers/infiniband/core/ib_ucm.ko

    ./mlnx-ofa_kernel/drivers/infiniband/core/ib_uverbs.ko

    ./mlnx-ofa_kernel/drivers/infiniband/core/ib_sa.ko

    ./mlnx-ofa_kernel/drivers/infiniband/core/iw_cm.ko

    ./mlnx-ofa_kernel/drivers/scsi/scsi_transport_srp.ko

    ./mlnx-ofa_kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko

    ./mlnx-ofa_kernel/drivers/scsi/cxgbi/libcxgbi.ko

    ./mlnx-ofa_kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko

    ./mlnx-ofa_kernel/drivers/net/ethernet/mellanox/mlx4_vnic/mlx4_vnic.ko

    ./mlnx-ofa_kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko

    ./mlnx-ofa_kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko

    ./mlnx-ofa_kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko

    ./mlnx-ofa_kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko

    ./mlnx-ofa_kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko

    ./mlnx-ofa_kernel/drivers/net/eipoib/eth_ipoib.ko

    ./mlnx-ofa_kernel/compat/compat.ko

    ./kernel-mft-mlnx/mst_pciconf.ko

    ./kernel-mft-mlnx/mst_pci.ko

    ./iser/ib_iser.ko

    ./srp/scsi/scsi_transport_srp.ko

    ./srp/ib_srp.ko

    iser/ib_iser.ko

    kernel-mft-mlnx/mst_pciconf.ko

    kernel-mft-mlnx/mst_pci.ko

    knem-mlnx/knem.ko

    mlnx-ofa_kernel/fs/9p/9p.ko

    mlnx-ofa_kernel/net/sunrpc/xprtrdma/svcrdma.ko

    mlnx-ofa_kernel/net/sunrpc/xprtrdma/xprtrdma.ko

    mlnx-ofa_kernel/net/rds/rds_tcp.ko

    mlnx-ofa_kernel/net/rds/rds.ko

    mlnx-ofa_kernel/net/rds/rds_rdma.ko

    mlnx-ofa_kernel/net/9p/9pnet_rdma.ko

    mlnx-ofa_kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko

    mlnx-ofa_kernel/drivers/infiniband/hw/nes/iw_nes.ko

    mlnx-ofa_kernel/drivers/infiniband/hw/mthca/ib_mthca.ko

    mlnx-ofa_kernel/drivers/infiniband/hw/qib/ib_qib.ko

    mlnx-ofa_kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko

    mlnx-ofa_kernel/drivers/infiniband/hw/ipath/ib_ipath.ko

    mlnx-ofa_kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko

    mlnx-ofa_kernel/drivers/infiniband/hw/amso1100/iw_c2.ko

    mlnx-ofa_kernel/drivers/infiniband/hw/cxgb3/iw_cxgb3.ko

    mlnx-ofa_kernel/drivers/infiniband/ulp/isert/ib_isert.ko

    mlnx-ofa_kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko

    mlnx-ofa_kernel/drivers/infiniband/ulp/iser/ib_iser.ko

    mlnx-ofa_kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko

    mlnx-ofa_kernel/drivers/infiniband/ulp/srp/ib_srp.ko

    mlnx-ofa_kernel/drivers/infiniband/core/rdma_ucm.ko

    mlnx-ofa_kernel/drivers/infiniband/core/ib_mad.ko

    mlnx-ofa_kernel/drivers/infiniband/core/ib_core.ko

    mlnx-ofa_kernel/drivers/infiniband/core/ib_cm.ko

    mlnx-ofa_kernel/drivers/infiniband/core/ib_addr.ko

    mlnx-ofa_kernel/drivers/infiniband/core/rdma_cm.ko

    mlnx-ofa_kernel/drivers/infiniband/core/ib_umad.ko

    mlnx-ofa_kernel/drivers/infiniband/core/ib_ucm.ko

    mlnx-ofa_kernel/drivers/infiniband/core/ib_uverbs.ko

    mlnx-ofa_kernel/drivers/infiniband/core/ib_sa.ko

    mlnx-ofa_kernel/drivers/infiniband/core/iw_cm.ko

    mlnx-ofa_kernel/drivers/scsi/scsi_transport_srp.ko

    mlnx-ofa_kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko

    mlnx-ofa_kernel/drivers/scsi/cxgbi/libcxgbi.ko

    mlnx-ofa_kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko

    mlnx-ofa_kernel/drivers/net/ethernet/mellanox/mlx4_vnic/mlx4_vnic.ko

    mlnx-ofa_kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko

    mlnx-ofa_kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko

    mlnx-ofa_kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko

    mlnx-ofa_kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko

    mlnx-ofa_kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko

    mlnx-ofa_kernel/drivers/net/eipoib/eth_ipoib.ko

    mlnx-ofa_kernel/compat/compat.ko

    srp/scsi/scsi_transport_srp.ko

    srp/ib_srp.ko

     

    Another example in case of Ubuntu 14.04

    # pwd

    /lib/modules/3.13.0-33-generic/updates/dkms

    # ls

    compat.ko

    ib_cm.ko

    ib_iser.ko

    ib_sa.ko

    ib_umad.ko

    knem.ko

    mlx4_ib.ko

    mst_pciconf.ko

    rdma_ucm.ko

    rds_tcp.ko

    eth_ipoib.ko

    ib_core.ko

    ib_isert.ko

    ib_srp.ko

    ib_uverbs.ko

    mlx4_core.ko

    mlx5_core.ko

    mst_pci.ko

    rds.ko

    scsi_transport_srp.ko

    ib_addr.ko

    ib_ipoib.ko

    ib_mad.ko

    ib_ucm.ko 

    iw_cm.ko     

    mlx4_en.ko   

    mlx5_ib.ko   

    rdma_cm.ko     

    rds_rdma.ko