eIPoIB Manual Configuration

Version 3

    Note: This document is outdated, and kept in archive section. Please refer to MLKNX_OFED User Manual for proper procedure.




    This document provides instructions for configuring eIPoIB interface manually.

    Normally eIPoIB interfaces are configured automatically by a user level

    service called /sbin/ipoibd, this service may not support some Operating

    Systems. The steps below describe how to bypass the ipoibd service and

    configure eIPoIB interface manually for Mellanox OFED-2.0-2.0.x.



    1. Configuration Check:


    Enable eIPoIB driver:

    > grep IPOIB /etc/infiniband/openib.conf




      E_IPOIB_LOAD=yes << This line enables eIPoIB



    Restart OFED stack:

    > /etc/init.d/openibd restart



    Stop ipoibd service:

    > pkill -9 ipoibd



    2. Driver Check:


    Make sure IPoIB is loaded:

    > lsmod | grep ipoib



    Make sure eIPoIB is loaded:

    > lsmod | grep eth_ipoib



    3. Logical Link Check:


    Check IB link:

    > ibstat |grep State

      State: Active



    Check IPoIB link (say ib0):

    > ifconfig ib0 up

    > cat /sys/class/net/ib0/carrier (1 indicates that the link is up):




    Check eIPoIB link (say eth2):

    > ifconfig eth2 up

    > cat /sys/class/net/eth2/carrier




    Find the eIPoIB network interface (go over ethX interfaces):

    > ethtool -i eth2

      driver: eth_ipoib << Look for driver name: eth_ipoib

      version: 1.0.0

      firmware-version: 1

      bus-info: ib0 << Underlying interface name (physical port #1)

      supports-statistics: yes



    4. Configure eIPoIB interface:


    Remember eIPoIB interface (eth2) MAC address:

    > ifconfig eth2 | grep HWaddr

      eth2      Link encap:Ethernet  HWaddr 00:1E:67:39:78:11



    Create a new IPoIB clone (say ib0.1):

    > echo .1 > /sys/class/net/ib0/create_child



    Check IPoIB clone (ib0.1) link status:

    > ifconfig ib0.1 up

    > cat /sys/class/net/ib0.1/carrier




    Enslave the IPoIB clone (ib0.1) to eIPoIB interface (eth2):

    > /bin/echo +ib0.1 > /sys/class/net/eth2/eth/slaves



    Enable eIPoIB interface (eth2) MAC address (use lower case):

    > /bin/echo +ib0.1 00:1e:67:39:78:11   > /sys/class/net/eth2/eth/vifs



    Confirm eth2 configuration:

    > cat /sys/class/net/eth2/eth/vifs

      SLAVE=ib0.1      MAC=00:1e:67:39:78:11 VLAN=N/A



    5. Configure eIPoIB Additional Interfaces:


    Some system configurations may require configuring eIPoIB interface to serve

    multiple interfaces such as a vif, tap, tun, br, etc.

    Below is an ASCII graphic description for a virtual network example:





                   |                     |

                   | ib0.2  ib0.3        |

                   |   |     |           |

                   |   +--+--+           |    

                   |      |              |   +----------------+

                   |  +---|--------+     |   |                |

      IPoIB        |  | eth2       |  +------------ vif0      |

       LAN -----+--|  |            |  |  |   |                |

    (via port #1)  |  | tap0 ---------+  |   +----------------+

                   |  | tap1 ---------+  |                          

                   |  |            |  |  |                        +--------------+

                   |  +------------+  |  |                        |              |

                   |       br0        +-------------------------------- tun0     |

                   |                     |                        |              |

                   +---------------------+                        +--------------+



    In order to configure eIPoIB interface to enable traffic for additional

    interfaces (for example, to let eth2 send/receive traffic for tun0/vif0);

    repeat step #4 for each one of the interfaces, e.g. if vif0 MAC address is:

    00:1E:67:39:78:22, then, in order let eth2 send/receive its traffic,

    run these commands:

    # Create a new ipoib clone

    > echo .2 > /sys/class/net/ib0/create_child

    # bring the ipoib clone logical link up

    > ifconfig ib0.2 up

    # enslave the new clone under eth2

    > /bin/echo +ib0.2 > /sys/class/net/eth2/eth/slaves

    # add the vif MAC to eth2 using the ib0.2 clone

    > /bin/echo +ib0.2 00:1e:67:39:78:22   > /sys/class/net/eth2/eth/vifs

    # confirm the configuration:

    > cat /sys/class/net/eth2/eth/vifs


      SLAVE=ib0.2      MAC=00:1e:67:39:78:22 VLAN=N/A





    6. Virtual LAN (VLAN) Configuration:


    An Ethernet interface may have 8021Q VLAN configured. In this case, the eIPoIB

    driver can map VLAN to an InfiniBand Partition Key (PKEY) and simulate VLAN

    services on top of an InfiniBand fabric. Note that the VLAN to PKEY mapping

    must be 1:1 (multiple VLANs cannot map to the same PKEY).



    First, configure the Subnet Manager to create a new PKEY for IPoIB usage.

    For example, check opensm(8) man page for Partition Configuration. Upon

    successful PKEY creation, you should be able to create IPoIB child interface

    with the PKEY you've created. Example for PKEY 0x8012:

    > echo 0x8012 > /sys/class/net/ib0/create_child

    > ifconfig ib0.8012 up # confirm interface creation and link status



    Second, create eIPoIB interface for the desired VLAN, for example, if you wish

    to run Ethernet VLAN 22 traffic on InfiniBand PKEY 0x8012, create eth2.22:

    > vconfig add eth2 22

    > ifconfig eth2.22 up # confirm interface creation and link status



    Now, you can repeat step #4 for any additional interface to use VLAN 22 which

    is mapped to PKEY=0x8012. For example, if the additional interface has

    the MAC=00:1e:67:39:78:33, run the following commands:

    # Create a new ipoib clone #3 for pkey 0x8012

    > echo 0x8012.3 > /sys/class/net/ib0/create_child

    # bring the ipoib clone logical link up

    > ifconfig ib0.8012.3 up

    # enslave the new clone under eth2 (the VLAN is assigned in the next step)

    > /bin/echo +ib0.8012.3 > /sys/class/net/eth2/eth/slaves

    # add the interface MAC and VLAN to eth2 using ib0.8012.3 clone

    > /bin/echo +ib0.8012.3 00:1e:67:39:78:33 22  > /sys/class/net/eth2/eth/vifs

    # confirm the configuration:

    > cat /sys/class/net/eth2/eth/vifs


      SLAVE=ib0.8012.3   MAC=00:1e:67:39:78:33 VLAN=22