Network Adoption Scenarios for RoCE applications

Version 9

    This post is aimed to supply various non-green field scenarios for RoCE based application. For the simplicity, the networks described in this example are Ethernet L2 networks.

     

    References

     

    RDMA based applications require no drops in the network (loss-less network).

     

    Important Note: In case the network consists of more than one switch, all the intra-switch links should be enabled with PFC on the configured priority (e.g. 3).

     

    Scenario 1: Two Servers, Two Adapters, Two Ports, Two Networks

    This scenario assumes two different networks (IP subnet and separated VLAN) are running over the same network switches.

    In this case, we assume that the network consists of various servers and switches that do not run any RoCE application, and we want to add some additional new servers (such as application servers and storage servers) that do not communicate with the legacy servers.

    In this example:

    • VLAN 10 is configured on the legacy servers. Flow control may be enabled or not (this depends on your current network implementation) on VLAN 10.
    • While adding new servers, we create a new network (IP subnet) for the legacy servers - VLAN 20.
    • The new servers attached to VLAN 20 do not communicate with the legacy servers attached to VLAN 10.

     

    Configuration

    • Switch Configuration
      • Global Configuration
        • QoS should be enabled on the switch (if it was not enabled before). It doesn't affect all network ports without enabling QoS on the port
        • Enable with PFC on the required priority (e.g 3)
      • Interface
        • Configuration on the ports connected to the legacy servers under VLAN 10
          • No port configuration changes are required
        • The switch configuration on the ports connected to the new servers under VLAN 20
          • Enabled with VLAN 20 (trunk)
          • Global pause flow control should be disabled
          • QoS on this port is enabled
    • Legacy Server Configuration
      • Nothing is required
    • New server configuration
      • QoS should be enabled on the servers ports connected to the switch
      • PFC should be enabled on those ports on the desired priority (e.g. 3)
      • RDMA should be configured properly based on the application and OS (e.g. SMB direct for Windows, iSER tgt for Linux RHEL7 etc.)
      • RDMA traffic should be mapped to the configured priority

    After this configuration, the network will be ready to carry the RDMA traffic without interfering with the legacy traffic

     

     

     

    1%3Fauth_token%3Dc7b5e77df74a8de8018da1bdd7449e4159fd154d

    Scenario 2: One Server, Two Adapters, Two Ports, Two Networks

    This scenario assumes two different networks (IP subnet and separated VLAN) are running over the same network switches.

    In this case, we assume that the network consists of various servers and switches that do not run any RoCE application, and we want to add some additional new adapters.

    After the adapter installation, each server will have two cards:

    • Legacy adapter card is not running RDMA
    • New adapter card is running RDMA

     

    In this example:

    • VLAN 10 is configured for the legacy adapter port. Flow control may be enabled or not (this depends on your current network implementation) on VLAN 10.
    • While adding new adapter cards, we create new network (IP subnet) for the legacy servers - VLAN 20.
    • The applications running on the servers attached to VLAN 20 do not communicate with the application attached to VLAN 10 (different VLAN - different application)

     

    Configuration:

    • Switch Configuration
      • Global Configuration
        • QoS should be enabled on the switch (if it was not enabled before). It doesn't affect all network ports without enabling QoS on the port
        • Enable with PFC on the required priority (e.g 3)
      • Interface
        • Configuration on the ports connected to the servers under VLAN 10
          • No port configuration changes are required
        • The switch configuration on the ports connected to the servers under VLAN 20
          • Enabled with VLAN 20 (trunk)
          • Global pause flow control should be disabled
          • QoS on this port is enabled
    • Server configuration
      • QoS should be enabled on the servers ports connected to the switch
      • PFC should be enabled on those ports on the desired priority (e.g. 3)
      • RDMA should be configured properly based on the application and OS (e.g. SMB direct for Windows, iSER tgt for Linux RHEL7 etc.)
      • RDMA traffic should be mapped to the configured priority

    After this configuration, the network will be ready to carry the RDMA traffic without interfering with the legacy traffic


    1%3Fauth_token%3D309c098649bd1e92c33a64b22ee1cb76f501b65b

     

     

    Scenario 3: One Server, One Adapter, One Port, Two Networks

    This scenario assumes two different networks (IP subnet and separated VLAN) are running over the same network switches.

    In this case, we assume that the network consists of various servers and switches that do not run any RoCE application, and we want to replace the legacy adapters with new adapters that support RDMA.

    After the adapter installation, each server will have one adapter card that supports RDMA.

     

    In this example:

    • VLAN 10 is configured for the legacy adapter port. Flow control may be enabled or not (depends on your current network implementation) on VLAN 10.
    • While replacing the adapter card, we create new network (IP subnet) for the legacy adapter - VLAN 20 on the same network interface.
      • In the case of Windows OS, a VEA interface should be created and not teaming.
      • In the case of any Linux OS, two VLAN interfaces should be created.
    • The applications running on the servers attached to VLAN 20 do not communicate with the application attached to VLAN 10 (different VLAN - different application)

     

    Configuration:

    • Switch Configuration
      • Global Configuration
        • QoS should be enabled on the switch (if it was not enabled before). It doesn't affect all network ports without enabling QoS on the port.
        • Enable with PFC on the required priority (e.g 3).
      • Interface
        • Enabled with VLAN 20 and VLAN 20 (trunk)
        • Global pause flow control should be disabled
        • QoS on this port is enabled
    • Server Configuration
      • QoS should be enabled on the servers ports connected to the switch
      • PFC should be enabled on those ports on the desired priority (e.g. 3)
      • RDMA should be configured properly based on the application and OS (e.g. SMB direct for Windows, iSER tgt for Linux RHEL7 etc.)
      • RDMA traffic should be mapped to the configured priority

    After this configuration, the network will be ready to carry the RDMA traffic without interfering with the legacy traffic.

    1%3Fauth_token%3D8f3538747a099845d6875efa310bc25bcca2f907

     

    Scenario 4: One Server, One Adapter, One Port, One Network

    This scenario assumes one network (IP subnet and separated VLAN) are running over the same network switches.

    In this case, we assume that the network consists of various servers and switches that do not run any RoCE application, and we want to replace the legacy adapters with new adapters that support RDMA.

    After the adapter installation, each server will have one adapter card that supports RDMA.

     

    In this example:

    • VLAN 10 is configured for the legacy adapter port. Flow control may be enabled or not (depends on your current network implementation) on VLAN 10.
    • While replacing the adapter card, keep both RDMA and TCP applications to operate over the same network using VLAN 10.

     

    Configuration:

    • Switch Configuration
      • Global Configuration
        • QoS should be enabled on the switch (if it was not enabled before). It doesn't affect all network ports without enabling QoS on the port
        • Enable with PFC on the required priority (e.g 3)
      • Interface
        • Enabled with VLAN 10 (trunk)
        • Global pause flow control should be disabled
        • QoS on this port is enabled
    • Server Configuration
      • QoS should be enabled on the servers ports connected to the switch
      • PFC should be enabled on those ports on the desired priority (e.g. 3)
      • RDMA should be configured properly based on the application and OS (e.g. SMB direct for Windows, iSER tgt for Linux RHEL7 etc.)
      • RDMA traffic should be mapped to the configured priority

    After this configuration, the network will be ready to carry the RDMA traffic without interfering with the legacy traffic

     

    1%3Fauth_token%3Dd7c522f84276efb91246315c59a7a0d15d240ead