HowTo Change Port Type in Mellanox ConnectX-3 Adapter

Version 11

    This post describes how to change the port type (eth, ib) in Mellanox adapters when using MLNX-OFED or Inbox drivers.

    Mellanox ConnectX®-3 adapter card (VPI) may be equipped with one or two ports that may be configured to run InfiniBand or Ethernet.

     

    By default, port configuration is set to ib.

     

    Note: in case you are using ConnectX-4 or later. See Getting started with ConnectX-4 100Gb/s Adapter for Linux, it is not the same procedure.

     

    References

     

    Configuration

    MLNX_OFED

    When using MLNX_OFED, port type can be configured for each device in the system at run time using the script "/usr/sbin/connectx_port_config" .

     

    For example:

    # connectx_port_config

     

    ConnectX PCI devices :

    |----------------------------|

    | 1             0000:82:00.0 |

    |----------------------------|

     

    Before port change:

    auto (ib)

    auto (ib)

     

    |----------------------------|

    | Possible port modes:       |

    | 1: Infiniband              |

    | 2: Ethernet                |

    | 3: AutoSense               |

    |----------------------------|

    Select mode for port 1 (1,2,3): 2

     

    Running “connectx_port_config -s”  will show current port configuration for all ConnectX devices.

     

    The options are:

    • InfiniBand (1)
    • Ethernet (2)
    • AutoSense (3) - Link sensing mode. It detect the port type based on the attached network type (InfiniBand or Ethernet) automatically and load the appropriate driver stack (InfiniBand or Ethernet). If no link is detected, the driver retries link sensing every few seconds.

     

    Port configuration is saved in the file:  /etc/infiniband/connectx.conf.

     

    This saved  configuration is restored at driver restart only if restarting via  “/etc/init.d/openibd  restart”.

     

    Use '-help' flag to see the command usage:

    # connectx_port_config --help

    Usage:

    /sbin/connectx_port_config

    /sbin/connectx_port_config -s|--show

    /sbin/connectx_port_config -h|--help

    /sbin/connectx_port_config [-d|--device <PCI device id>] -c|--conf <port1,port2>

     

    All possible ports configuration combinations:

      ib,ib

      ib,eth

      ib,auto

      eth,ib

      eth,eth

      eth,auto

      auto,ib

      auto,eth

      auto,auto

     

    Inbox Driver

     

    Options 1:

    In order to change the port mode without the script (Inbox support):

     

    echo “mode” >/sys/bus/pci/devices/”pci device”/mlx4_port1

    echo “mode” >/sys/bus/pci/devices/”pci device”/mlx4_port2

     

     

    For Example:

     

    # lspci | grep Mellanox

     

    01:00.0 InfiniBand: Mellanox Technologies MT26428 [ConnectX IB QDR, PCIe 2.0 5GT/s] (rev b0)

     

    #echo ib    > /sys/bus/pci/devices/0000\:01\:00.0/mlx4_port1

    #echo eth > /sys/bus/pci/devices/0000\:01\:00.0/mlx4_port2

     

    Note: This option is not persistent.

     

    Option 2 (in case RDMA is in use):

     

    Edit the file /etc/rdma/mlx4.conf:

    Note: This file is read when the mlx4_core module is loaded and used to set the port types for any hardware found.

     

    Format:

    <pci_device_of_card> <port1_type> [port2_type]

     

    port1 and port2: One of "auto", "ib", or "eth". port1 is required at all times, port2 is required for dual port cards.

     

    For example:

    0000:05:00.0 eth eth

    Perform reboot to reload the modules

    #reboot

     

    MFT

    Use Mellanox Firmware Tools (MFT) to change the firmware configuration of the port. See some examples in Getting Started with ConnectX-5 100Gb/s Adapters for Linux.

    This example suit ConnectX-3  as well.