9 Replies Latest reply on Jun 17, 2013 12:43 PM by ecosse

    Very stupid MHGH28-XTC question


      Hi

       

      Can you load an Ethernet driver for these cards?  I've bought a couple of Sun badged ones and flashed them back to Mellanox firmware.  As an IB card it looks fine but I really wanted Ethernet support so I could attach it to an Ethernet switch.  There are some drivers in the package but loading them results in a "code 10" within Windows (2012).  I've tried the 2008r2 drivers with similar results. So just asking if I am being particularly thick here before I troubleshoot more!

       

      Thanks
      Drew

        • Re: Very stupid MHGH28-XTC question

          You need the ConnectX3 Cards for the right firmware to work on Server 2012.  The ConnectX2 cards are not supported.

           

          Mike

            • Re: Very stupid MHGH28-XTC question

              Thanks for the reply.  Of course, not supported does not necessarily mean it won't work i.e. I've got a few Connect X MNEH cards working perfectly fine in Windows 2012.

               

              Anyway, are you saying that these will work fine on Windows 2008R2?  I guess I'll check now.

                • Re: Very stupid MHGH28-XTC question

                  Some of the ConnectX Cards will work, it depends on the firmware, but they are not supported for Windows 2012.  Some of the RDMA features won't/don't work.  I have not used them with Windows Server 2008R2.

                    • Re: Very stupid MHGH28-XTC question

                      Hi Mike,

                       

                      Thanks but kind of know all you've posted already.  I'll see what happens on Windows 28kr2 / Centos I guess.

                       

                      Thanks

                      Drew

                        • Re: Very stupid MHGH28-XTC question
                          inbusiness

                          Hi~

                           

                          If you have a ConnectX-2 VPI firmware above 2.9.8350 then you can use it!

                          If Mellanox support ConnectX-2 EN INTERIM firmware you can use it on WS2k12.

                           

                          Good luck!

                          • Re: Re: Very stupid MHGH28-XTC question
                            justinclift

                            ecosse - You can definitely make it work with CentOS 6.x.

                             

                            With (for example) CentOS 6.4 using the CentOS supplied packages (yum groupadd "Infiniband Support"), the MHGH28-XTC cards can run in either Infiniband or native 10GbE mode.  By default the cards are in Infiniband mode.

                             

                            The file used to configure which mode is:

                             

                            /etc/rdma/mlx4.conf

                             

                            The comments in the file give decent instructions on what to do.

                             

                            As a helpful reference example, this shows the PCI address for one of my MHGH28-XTC cards in a CentOS 6.4 test box here:

                            $ lspci | grep Mellanox

                            01:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX VPI PCIe 2.0 2.5GT/s - IB DDR / 10GigE] (rev a0)

                            That "01:00.0" address is then used in the mlx4.conf file as:

                            0000:01:00.0 eth eth

                            After rebooting, the cards show up as native ethernet cards:

                            $ ifconfig
                            eth1      Link encap:Ethernet  HWaddr 00:03:BA:00:ED:BA  
                                      inet addr:172.16.101.1  Bcast:172.16.101.255  Mask:255.255.255.0
                                      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
                                      RX packets:0 errors:0 dropped:0 overruns:0 frame:0
                                      TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
                                      collisions:0 txqueuelen:1000 
                                      RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
                            
                            eth2      Link encap:Ethernet  HWaddr 00:03:BA:00:ED:B9  
                                      inet addr:172.16.102.1  Bcast:172.16.102.255  Mask:255.255.255.0
                                      UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
                                      RX packets:0 errors:0 dropped:0 overruns:0 frame:0
                                      TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
                                      collisions:0 txqueuelen:1000 
                                      RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

                            and:

                            $ ip addr 
                            2: eth2:  mtu 1500 qdisc mq state UNKNOWN qlen 1000
                                link/ether 00:03:ba:00:ed:b9 brd ff:ff:ff:ff:ff:ff
                                inet 172.16.102.1/24 brd 172.16.102.255 scope global eth2
                            3: eth1:  mtu 1500 qdisc mq state UNKNOWN qlen 1000
                                link/ether 00:03:ba:00:ed:ba brd ff:ff:ff:ff:ff:ff
                                inet 172.16.101.1/24 brd 172.16.101.255 scope global eth1

                            The ethernet configuration of them is then done through the standard files in /etc/sysconfig/network-scripts/.

                             

                            As a helpful reference these are two working config files (for the MHGH28-XTC card in the above example):

                            $ cat /etc/sysconfig/network-scripts/ifcfg-eth1

                            DEVICE=eth1

                            TYPE=Ethernet

                            ONBOOT=yes

                            NM_CONTROLLED=no

                            BOOTPROTO=none

                            DEFROUTE=no

                            PEERDNS=no

                            PEERROUTES=no

                            IPV4_FAILURE_FATAL=yes

                            IPV6INIT=no

                            NAME="System eth1"

                            IPADDR=172.16.101.1

                            PREFIX=24

                            and:

                            $ cat /etc/sysconfig/network-scripts/ifcfg-eth2

                            DEVICE=eth2

                            TYPE=Ethernet

                            ONBOOT=yes

                            NM_CONTROLLED=no

                            BOOTPROTO=none

                            DEFROUTE=no

                            PEERDNS=no

                            PEERROUTES=no

                            IPV4_FAILURE_FATAL=yes

                            IPV6INIT=no

                            NAME="System eth2"

                            IPADDR=172.16.102.1

                            PREFIX=24

                             

                            Hope that all gets you up and running.

                              • Re: Re: Very stupid MHGH28-XTC question
                                justinclift

                                Ugh, the Jive commenting system seems to be messing up the formatting of the above post.  It looks fine in the advanced editor, but is screwy when actually displayed.

                                 

                                The info in it is still all correct however.

                                  • Re: Very stupid MHGH28-XTC question

                                    Hi Justin

                                     

                                    Many thanks for the detailed information - really appreciated.  I really need my primary machine running under Windows so I guess I need to mess around with W2K8R2 (but its a shame if I have to regress the OS to get it to work - just bought a Myricom 10gb card just in case) but I have a spare card or two so I'm going to brush up on my not very good Centos skills and give your instructions a bash!

                                    Cheers

                                    Drew

                                      • Re: Very stupid MHGH28-XTC question

                                        Just in case anyone had the same issue as me, the Connect-X Windows 7 x64 drivers seem to work fine in Windows Server 2012 using this card.  I've pinged and copied a few files around so hopefully when I setup my storage it keeps up!

                                         

                                        Maybe I downloaded the wrong W2K8R2 ones but even though a comparison of versions were correct in control panel the W2K8R2 drivers were dated 2009 in  the program files location whereas the Windows 7 driver set was some time in 2010. 

                                        Cheers

                                        Drew