All about VPI, Gateway and Hybrid networks

Version 4

    This post describes VPI operation and discusses Ethernet to InfiniBand Gateway processes at high level.

    This post is basic and meant for beginners.

     

     

    References

     

    Overview

    Products based on the Mellanox ConnectX family of adapters and Mellanox SwitchX® family of switches support Virtual Protocol Interconnect (VPI). VPI allows InfiniBand and Ethernet traffic to co-exist on one platform. Each port can operate independently as an InfiniBand link or as an Ethernet link. ConnectX adapters can be configured to use InfinBand, Ethernet or to auto-sense the fabric through the port itself. In addition, a gateway can be added between Ethernet and InfiniBand subnets to link servers on the same subnet.

    cluster-highlights-marketing-slide.png

    Mellanox Adapters

    ConnectX adapter family ports can be configured with the following protocol link types:

    • Ethernet (eth) – sets the port as an Ethernet link
    • InfiniBand (ib) – sets the port as an InfiniBand link
    • Auto Sensing (auto) – in this mode the port detects the port type based on the attached network type.

     

    Setting the link protocol on ports of the ConnectX adapter family can be performed using the MLNX_OFED (Linux) or WinOF (Windows) drivers. For easy migration from InfiniBand to Ethernet and vice versa, configure your adapter cards to Auto Sensing mode. Auto Sensing mode enables the adapter card to automatically sense the link type (InfiniBand or Ethernet) based on the cable connected to the port and load the appropriate driver stack (InfiniBand or Ethernet). If no link is detected, the driver retries link  sensing every few seconds. For example, if the first port is connected to an InfiniBand switch and the second to an Ethernet switch, the adapter card automatically loads the first port as InfiniBand and the second as Ethernet.

     

    Mellanox Switches

    The operation of VPI technology on Mellanox switch systems can be achieved in different levels:

    • System level VPI – it can be decided, per system, whether to use InfiniBand or Ethernet for all the interfaces of the system. Either an Ethernet switch or an InfiniBand switch profile can be configured on the system in order to determine the running link protocol for all the system ports.
    • Interface level VPI – it can be decided, per system port, whether to use InfiniBand or Ethernet as a link protocol. A single VPI system profile can be configured and, per port, the link protocol may be selected. Configuring the switch to VPI mode by interface allows splitting the hardware into two separate switches (an Ethernet switch and an InfiniBand switch). Traffic does not pass between those switches. While configuring the VPI system profile, bridging (or gateway) capabilities can be added to pass traffic from the Ethernet to the InfiniBand hosts.

     

    Gateway

    Gateway is basically a switch in the network, where some of its ports are configured as Ethernet while others as InfiniBand. For a hybrid network of Ethernet and InfiniBand, Proxy-ARP is used to forward IPv4 packets from the Ethernet network to the InfiniBand network and vice versa. Proxy-ARP is not an IP Router; rather, it acts as a bridge that forwards the IPoETH packets to IPoIB in Unreliable Datagram (UD) mode. The Proxy-ARP forwards the traffic within a single subnet.

     

    Gateway.png

    ARP Flow

    An ARP request sent from an InfiniBand host is terminated at the Proxy-ARP function on the gateway. Then a new ARP request is generated and sent on the VLAN interface to reach the Ethernet host. The Ethernet host responds with an ARP reply to the Proxy-ARP. The Proxy-ARP function then terminates it and generates a new ARP reply to the InfiniBand host. Once the destination address has been resolved, unicast traffic is passed from the InfiniBand host to the Ethernet host. The process is similar in the opposite direction (Ethernet to InfiniBand).

    ARP flow.png

    Refer to VPI Gateway Considerations for deeper discussions about gateways and network topologies.

     

    Cables Considerations

    There are no specific cable considerations. Each port should be connected to a cable that supports the required data rate and protocol.

     

    Configuration

     

    Mellanox Adapters (Linux/Windows)

    In order to change the port type via the MLNX_OFED (Linux) driver, refer to HowTo Change Port Type in Mellanox ConnectX-3 Adapter.

    In order to change the port type via the WinOF (Windows) driver, refer to HowTo Install and Configure Mellanox Driver for Windows Environment.

     

    Gateway Configuration

     

    A comprehensive description of Mellanox Gateway configuration can be found in the MLNX-OS User Manual. The following solutions are also available in case they are suitable for your network: