4 Replies Latest reply on Apr 26, 2015 3:49 AM by cloudbuilder

    Difference between PRO VPI and PRO EN adapter (RoCE V2) (Hyperconverged Network, SwitchX connected to internet?)

    cloudbuilder

      What exactly is the difference between the PRO VPI and PRO EN adapter? Can the PRO EN adapter do RoCE V2? Are there any adapters other than the VPI PRO that can do RoCE V2?

       

      Btw, My network is small (max 30 nodes), what benefits would RoCE V2 have over RoCE v1 for me? Is the routing only handy when working in large environments?

       

      Are there any users that use their mellanox switchx as the all-in-one switch/router? Can you connect the WAN directly to the switch, and use it with the same functionality as a normal router?

       

      Would 2 SX1036 switches be all the switching hardware required for small (sub 30 nodes) hyper-converged clusters? What other things would you advice on getting?

       

      Thanks

        • Re: Difference between PRO VPI and PRO EN adapter (RoCE V2) (Hyperconverged Network, SwitchX connected to internet?)
          ophirmaor

          The VPI (ConnectX-3 and ConnectX-3 Pro) cards can run InfiniBand on the link layer, while the EN cards cannot.

          RDMA is supported on both types (VPI and EN). if you use EN you can run RoCE.

          The ConnecX-3 Pro (EN or VPI) can run RoCE v.2 (routable) while the ConnecX-3 cannot.

           

          If you don't need to pass router, you may be good with RoCE v1 setup (not rotatable).

           

          What is your WAN connectivity? is it 10GbE?

          What about security?

           

          The SX1036 can be configured as a router as well.

          What L2/L3 protocols are you running?

           

          Thanks,

          Ophir.

            • Re: Difference between PRO VPI and PRO EN adapter (RoCE V2) (Hyperconverged Network, SwitchX connected to internet?)
              cloudbuilder

              Are you sure the connect-3 PRO EN can also do RoCE v2 ? The documentation only says roce , not roce v2! and the VPI PRO documentation clearly states RoCE V2!

               

               

              I have 2x 1gb WAN links, These are fiber, and I would like to avoid UTP/Base-T connections because I want the best latency possible. So, I think I'll need QSFP to SFP+ for that right? Or would it be possible to connect the fiber directly to the switch?

               

              about security: I was hoping to do all security with SDN software, any tips?

               

              another important question; Can I simultaneously use FcoE, RoCE V2 and ISER without reconfiguring the switch? (Each protocol in a separate Vlan ofcourse!!)

               

              Any advice on using just 2x1036 as my only networking hardware?

               

              Hope to hear from you soon!

                • Re: Difference between PRO VPI and PRO EN adapter (RoCE V2) (Hyperconverged Network, SwitchX connected to internet?)
                  ophirmaor

                  Hi Bent,

                  ConnecX-s Pro EN or VPI support RoCE v2 (routable).

                  Could you point me to the documentations you saw about that.

                   

                  SX1036 supports 1GbE (configurable). You will need to use QSA (40->10G adapter) and 10G->1G module.

                   

                  About security, it is too big subject. it is very hard to recommend here.

                  I assume that some companies would use special hardware for that (e.g. CheckPoint)

                  Mellanox switches supports ACL (Access control) so you can limit the access to specific MACs/IPs if that helps.

                  In some cases I assume that the ISP supply some security services as well.

                   

                  I don't see a problem with running FCoE and RoCE (v1 or v2) on the same switch.

                  you will need to enable PFC on the right priority. it could be the same priority or different priority for each of them.

                  Besides that the RoCE/FCoE layer is transparent to the switch.

                  Do you plan to use the switch as L3 router? how many subnets do you have in your planed network?

                   

                  How many ports do you want to use in each server connected to the switches ? two? one for each switch?

                  It will be helpful if you could send/attach a network diagram.

                   

                  Ophir.

                    • Re: Difference between PRO VPI and PRO EN adapter (RoCE V2) (Hyperconverged Network, SwitchX connected to internet?)
                      cloudbuilder

                      Well, I am not really sure about allot of things.

                       

                      This is what I have;

                       

                      3 Servers with ConnectX3 VPI PRO adapters 56gbe dual port
                      3 Servers with ConnectX3 EN adapters 56gbe dual port
                      2x sx1036 with following licenses: 56gbe + L3

                       

                      I want maximum performance(Load Balancing?), and redundancy for my network. And I want to keep things as simple as possible by only using these switches for all of my networking.
                      Each server is connected with one link to each switch. I guess I should also link the switches to eachother right? For loadbalancing and routing etc.?

                       

                      As of now I have one subnet, but this could expand in the future, nothing crazy though,
                      Im still fighting with different ISP's for the best deal and hardware, but most of them ship a FTU/Modem with 1x 1gb RJ45 port. (Im starting at 500/500 bandwith) This gets me to the following questions;

                      1: What would be the most effective way to connect my SX1036's to the RJ45(1g) port while minimizing added latency?
                      2: How much of a difference would a direct SFP+ connection make? Is this something worth fighting for?
                      3: Would it be possible to pass the whole modem/FTU and put each of the pair of(1 up 1 down) fibers in a SFP+ connector, and connect both these SFP+ connectors to the splitout cable to one of the switches? Or would this pose a problem with the SX1036 because of the wan link consisting of 2 ports? (one up one down). Is it even possible to do this kind of thing with the SX1036 and SX6036 series? Or do you need special tech for this kind of thing?

                       

                      So, I absolutely plan to use both switches as L3 router, while also using them in L2, and actually as my all-in-one networking solution:) Which ofcourse, still needs a firewall;

                      Mellanox says the SX1036 is capable of SDN, but this is very restricted it seems? Are there any firewalls that can run on these boys? What kind of SDN DOES work on these switches?
                      If firewall on the switches is not possible, a VM serving as a firewall on each host would be the next best thing right? No, I think dedicated would prob. be easier, but buying 2 ded. switches adds allot of costs, and also ads latency, so because of that I think using VM's with SR-IOV instead. Would a linux based firewall runing inside a VM using SR-IOV and proper QoS tuning be faster than a dedicated FW? If not, How much latency will this add?

                       

                      After my internet is connected to my host, and behind a firewall, I need to get it to the running vm's, what would be the lowest-latency way of doing this? I'm sure leveraging the adapters capabilities can help allot here? I was thinking SR-IOV ? Or would this not work with a FW VM, since everyone connects to the connector "directly" using sriov?

                       

                      Any thoughts on how to solve the FW problem? Do I even need a dedicated firewall? Cant the switches and adapters do most of the FW work? Combined with NVGRE/VXLAN tunneling?

                       

                      I would really appreciate any advice on both matters.

                       

                      Thank you,

                       

                      Builder