Dell Storage with Microsoft Storage Spaces (DSMS) - Networking Quick Reference

Version 19

    This post describes examples on how to enable Dell Storage with Microsoft Storage Spaces (Dell PowerEdge Storage servers with Microsoft SMB Direct) using Mellanox ConnectX-3/ConnectX-3 Pro adapters. This post assumes the reader has prior knowledge of their networking environment and Mellanox adapters.

     

    PowerEdge R730

    DELL PowerEdge R730.jpg

    Dell S4810

    s4810.jpg

     

    References

    General

    Configuration

    Testing

     

    Overview

    There are various of ways to deploy Microsoft SMB direct (RDMA) for Dell Storage with Microsoft Storage Spaces (DSMS) using Mellanox ConnectX-3/ConnectX-3 Pro adapters. Regarding networking, the main issue the IT architect should consider is whether there are more than one traffic flows coming from the same port on the server. In most basic out of the box cases, the setup will include only the type of network (on subnet) with one type of flow that consist of SMB direct over RDMA.

     

    The discussion about RDMA and what are the network requirement for RDMA is out of the scope of this post (Refer to What is RDMA?, Network Considerations for Global Pause, PFC and QoS with Mellanox Switches and Adapters, and Network Adoption Scenarios for RoCE applications).

     

    There are two options to create lossless L2 networks  for RoCE:

    • Using Flow Control (FC) - port based (global pause)
    • Using Priority Flow Control (PFC) - L2 priority based

     

    In case there is only one traffic type coming out of the network port, the usage of global FC in most cases will be sufficient. However, in case there are several traffic types (for example, RDMA and TCP) coming out of the same network port, PFC is recommended.

     

    Setup

    A basic setup should include:

    • One client server equipped with Mellanox ConnectX-3/ConnectX-3 Pro adapter
    • One S4810 Dell switch (or other DSMS supported switch)
    • One storage node server (or other DSMS supported config) equipped with Mellanox ConnectX-3/ConnectX-3 Pro adapter
    • 10/40GbE cables

     

    1%3Fauth_token%3D35c7b99c030ae75d28cc846898488e230f621c99

     

    Network Configuration

     

    Enabling RoCE in Windows Server 2012

    Mellanox ConnectX-3 adapters, use RoCEv1 as the RoCE mode configuration:

    PS c:\> Set-MlnxDriverCoreSetting –RoceMode 1

    Mellanox ConnectX-3 Pro adapters, use RoCEv2 as the RoCE mode configuration:

    PS c:\> Set-MlnxDriverCoreSetting –RoceMode 2

    Network Configuration based on Global Flow Control

    In this basic example, there is only one network (one VLAN) and one RDMA flow running between the client and the server.

    1. Follow HowTo Configure RoCE in Windows Environment (Global Pause) example to setup the client node and the PowerEdge storage servers.

     

    2. Configure flow control on the Dell S4810 switch on the relevant ports:

    # flowcontrol rx on tx on

     

    3. Verify that RDMA is configured properly by using nd_send_bw as mentioned in HowTo Configure RoCE in Windows Environment .

    You can also use RamDisk Ram Disk Application for Windows Environment (imdisk, sqlio) on the server side (before the configuration of the disks).

     

    Network Configuration based on PFC

    In this more complex example, there could be several types of flows (e.g. RDMA for stroage and TCP for web services) running over the same network (one VLAN) using the Mellanox ConnectX-3/ConnectX-3 Pro adapter.

    1. For Mellanox ConnectX-3 adapters (RoCEv1), useHowTo Configure SMB Direct (RoCE) over PFC on Windows 2012 Server to setup the client and storage server nodes.


    2. For Mellanox ConnectX-3 Pro adapters (RoCEv2), use HowTo Configure SMB Direct over IP networks (RoCEv2) on Windows 2012 Server to setup the client and storage server nodes.

     

    3. Configure PFC on the Dell S4810 (or other DSMS supported switch) on the relevant ports, refer to HowTo Enable PFC on Dell S6000/S4810 , for the configuration process.

     

    4. To verify the setup refer to HowTo Configure SMB Direct (RoCE) over PFC on Windows 2012 Server that uses sqlio and Ramdisk Ram Disk Application for Windows Environment (imdisk, sqlio) .

     

    DSMS Storage Configuration

    Once you verified that SMB direct (using RDMA) is running over the network (while using FC or PFC), you can continue and configure the storage volumes on the PowerEdge server. Please refer to the DSMS product support (http://dell.com/dsmsmanuals) page for further documentation.