6 Replies Latest reply on Mar 9, 2017 11:22 AM by fangchin

    Need to understand how MCX 4121A-ACAT card connect to 10 Gbps SPF+ connection

    rliu008

      Hi,

       

      We plan to install MCX 4121A-ACAT in our servers.  It has two ports thats should support 1/10/25/40/50 Gbps.  We need some advice how to hook up this card to the following:

       

      Customer SWITCH 10 Gbps SPF+ Connection <----10 Gbps SPF+------> MCX 4121A-ACAT <----25 Gbps SPF28---------> Mellanox 10/25/50 Switch

       

      Our customer might not have a SPF28 25 Gbps switch port.  They might only have SPF+ 10 Gbps switch port.  My question is the following:

       

      1) What kind of cable do I need to connect from MCX 4121A-ACAT card to the customer switch going from 25 Gbps SPF28 to SPF+ 10 Gbps?

       

      2) What kind of cable do I need on the connection to the Mellanox switch?

       

      3) Will there be performance degradation going from 25 Gbps to 10 Gbps?

       

      4) Is there a switch the forum can recommend that has both SPF28 and SPF+ ports to make inoperability between 25 Gbps and 10 Gbps seamlessly?

       

      Thanks for any help in advance.

       

      Thanks.

        • Re: Need to understand how MCX 4121A-ACAT card connect to 10 Gbps SPF+ connection
          faux_grey

          Hi Robert,

           

          1. Connecting the MCX-ACAT to the customer switch should be done with a normal 10G SFP+ cable, the Mellanox network adapter can then be set to speed 10G on that port.

          Item code: MC3309130-002

           

           

          2. Connecting the MCX-ACAT to a Mellanox 25G switch (spectrum series) can be done with a SFP28 / 25G DAC cable, please check with what switch you're wanting to connect, as some have SFP28 ports and some have QSFP28 ports, but both can be used.

          Item code: MCP2M00-A002

           

          3. Performance (throughput) will be degraded in terms of speed when running 10G instead of 25G (this much is obvious) but latency should remain the same.

           

          4. SFP28 ports can all be run at speeds of 25/10/1 G - No need to have both SFP28 and SFP+ ports, you only need SFP28.

          Depending on what your requirements are you could use the SN2410 which is the perfect top or rack switch, or you could use a SN2100 switch with hybrid/splitter cables for smaller scale deployments.

          1 of 1 people found this helpful
            • Re: Need to understand how MCX 4121A-ACAT card connect to 10 Gbps SPF+ connection
              rliu008

              Hi Faux,

               

              Thank you so much for all the great information.  It is really helpful.

               

              1.  It looks like all it takes to connect the card is a normal TWINAX cable.  Mellanox MC3309130-002 seems to do the trick, just needs the right length.  When you say MCX-ACAT needs to be at 10 Gbps, do you mean it will be done automatically or is there something we have to do in driver/software?  Shouldn't the MCX-ACAT automatic do this at the physical level?

               

              4.  SN2410 switch seems like the right fit for what we need.  We were going to deploy SX1012 switch.  Will SX1012 switch be compatible with MCX-ACAT above?  It seems SN2410 might be the better fit for future proof?

               

              Robert

                • Re: Need to understand how MCX 4121A-ACAT card connect to 10 Gbps SPF+ connection
                  faux_grey

                  Hi Robert,

                   

                  Twinax/DAC, really just a term for the cable with SFP endpoints.

                   

                  The Adapter will pick up that the cable inserted is 10G and should negotiate to that speed, all mellanox cables and transceivers have a small identification chip in them which can be seen by the hardware.

                   

                  The SX1012 switch is a older standard 1/10/40G switch, while the MCX-ACAT adapter card is a 1/10/25G Card, if you used them together, the highest speed you would be able to get is 10G.

                  If you want 25G from the card, you'll need a 25G compatible switch, which are any of the Spectrum SN switches.

                   

                  If you do purchase a SX1012 switch, I would rather recommend you take the ConnectX 3 Pro adapters, as they both support 1/10/40G.

                   

                  If you want 25G You need a ConnectX 4 LX adapter like the one we talked about above, and a Spectrum SN series switch.

                   

                  I must say, THe price difference between the older SX switches and the newer SN switches are completely justified, as you are future-proofed and won't have to upgrade again as soon.

                    • Re: Need to understand how MCX 4121A-ACAT card connect to 10 Gbps SPF+ connection
                      fangchin

                      Hi Faux,

                       

                      This is interesting.  We have

                      • Mellanox 25G DACs
                      • Mellanox 10G DACs
                      • Arista 10G DACs
                      • Dell 10G DACs
                      • Some third party (meaning inexpensive 10G DACs. 

                       

                      By saying below, did you mean the SN2410 can auto-detect the speed of the cable only if Mellanox cables are used?  Any pointer to Mellanox doc? 

                       

                      Faux Grey wrote:

                      [...]

                       

                      The Adapter will pick up that the cable inserted is 10G and should negotiate to that speed, all mellanox cables and transceivers have a small identification chip in them which can be seen by the hardware.

                       

                      [...]

                       

                      Thanks,

                       

                      Chin

                        • Re: Need to understand how MCX 4121A-ACAT card connect to 10 Gbps SPF+ connection
                          faux_grey

                          Hi Chin,

                           

                          I've never had any issues using 3rd party DAC cables or transceivers with Mellanox switches, but have never seen them auto-select cable speed. (I'm always manually forcing interface speeds)

                          Currently using some Arista, Plantronics, Finistar and prolabs cables/transceivers in my environment and they all work, but only the Mellanox ones set the correct speed without me configuring it. (using 10G transceivers in 25G port)

                           

                          As in the article below:

                          Introduction to LinkX DAC Cables

                          You can see that there is a Cable ID EPROM inside the LinkX cable - this allows Mellanox switches to Identify Mellanox modules and cable types (VPI/IB/ETH)

                            • Re: Need to understand how MCX 4121A-ACAT card connect to 10 Gbps SPF+ connection
                              fangchin

                              Hi Faux,

                               

                              Thanks for sharing your experience.

                              [...]

                               

                              I've never had any issues using 3rd party DAC cables or transceivers with Mellanox switches, but have never seen them auto-select cable speed. (I'm always manually forcing interface speeds)

                              Currently using some Arista, Plantronics, Finistar and prolabs cables/transceivers in my environment and they all work, but only the Mellanox ones set the correct speed without me configuring it. (using 10G transceivers in 25G port)

                               

                              [...]

                              This is my observation as well based on my limited trials.   Thanks for your confirmation.

                               

                              Chin