This post describes VPI operation and discusses Ethernet to InfiniBand Gateway processes at high level.
This post is basic and meant for beginners.
- HowTo Change Port Type in Mellanox ConnectX-3 Adapter
- HowTo Install and Configure Mellanox Driver for Windows Environment
- VPI Gateway Considerations
- HowTo Configure InfiniBand Gateway HA (Proxy ARP)
- HowTo Configure IPoIB Networks with Gateway and Multiple PKEYs
- HowTo Configure MLAG + Gateway HA
- MLNX-OS VPI User Manual
Products based on the Mellanox ConnectX family of adapters and Mellanox SwitchX® family of switches support Virtual Protocol Interconnect (VPI). VPI allows InfiniBand and Ethernet traffic to co-exist on one platform. Each port can operate independently as an InfiniBand link or as an Ethernet link. ConnectX adapters can be configured to use InfinBand, Ethernet or to auto-sense the fabric through the port itself. In addition, a gateway can be added between Ethernet and InfiniBand subnets to link servers on the same subnet.
ConnectX adapter family ports can be configured with the following protocol link types:
- Ethernet (eth) – sets the port as an Ethernet link
- InfiniBand (ib) – sets the port as an InfiniBand link
- Auto Sensing (auto) – in this mode the port detects the port type based on the attached network type.
Setting the link protocol on ports of the ConnectX adapter family can be performed using the MLNX_OFED (Linux) or WinOF (Windows) drivers. For easy migration from InfiniBand to Ethernet and vice versa, configure your adapter cards to Auto Sensing mode. Auto Sensing mode enables the adapter card to automatically sense the link type (InfiniBand or Ethernet) based on the cable connected to the port and load the appropriate driver stack (InfiniBand or Ethernet). If no link is detected, the driver retries link sensing every few seconds. For example, if the first port is connected to an InfiniBand switch and the second to an Ethernet switch, the adapter card automatically loads the first port as InfiniBand and the second as Ethernet.
The operation of VPI technology on Mellanox switch systems can be achieved in different levels:
- System level VPI – it can be decided, per system, whether to use InfiniBand or Ethernet for all the interfaces of the system. Either an Ethernet switch or an InfiniBand switch profile can be configured on the system in order to determine the running link protocol for all the system ports.
- Interface level VPI – it can be decided, per system port, whether to use InfiniBand or Ethernet as a link protocol. A single VPI system profile can be configured and, per port, the link protocol may be selected. Configuring the switch to VPI mode by interface allows splitting the hardware into two separate switches (an Ethernet switch and an InfiniBand switch). Traffic does not pass between those switches. While configuring the VPI system profile, bridging (or gateway) capabilities can be added to pass traffic from the Ethernet to the InfiniBand hosts.
Gateway is basically a switch in the network, where some of its ports are configured as Ethernet while others as InfiniBand. For a hybrid network of Ethernet and InfiniBand, Proxy-ARP is used to forward IPv4 packets from the Ethernet network to the InfiniBand network and vice versa. Proxy-ARP is not an IP Router; rather, it acts as a bridge that forwards the IPoETH packets to IPoIB in Unreliable Datagram (UD) mode. The Proxy-ARP forwards the traffic within a single subnet.
An ARP request sent from an InfiniBand host is terminated at the Proxy-ARP function on the gateway. Then a new ARP request is generated and sent on the VLAN interface to reach the Ethernet host. The Ethernet host responds with an ARP reply to the Proxy-ARP. The Proxy-ARP function then terminates it and generates a new ARP reply to the InfiniBand host. Once the destination address has been resolved, unicast traffic is passed from the InfiniBand host to the Ethernet host. The process is similar in the opposite direction (Ethernet to InfiniBand).
Refer to VPI Gateway Considerations for deeper discussions about gateways and network topologies.
There are no specific cable considerations. Each port should be connected to a cable that supports the required data rate and protocol.
Mellanox Adapters (Linux/Windows)
In order to change the port type via the MLNX_OFED (Linux) driver, refer to HowTo Change Port Type in Mellanox ConnectX-3 Adapter.
In order to change the port type via the WinOF (Windows) driver, refer to HowTo Install and Configure Mellanox Driver for Windows Environment.
A comprehensive description of Mellanox Gateway configuration can be found in the MLNX-OS User Manual. The following solutions are also available in case they are suitable for your network: