6 Replies Latest reply on Sep 14, 2017 10:35 PM by bgad15

    RoCE not working on Win 2016 (ConnectX-3 Pro)

    demonixed

      Hi all!

      I'm trying to configure SMB Direct on windows 2016 with ConnectX-3 Pro EN card (MCX312C-XCCT), but it's not working. RDMA Activity counters shows constantly rising Responder CQE Errors. So, I treied to buils simplest configuration and now I have the following

      Two servers with WIn 2016  w/latest updates. ConnectX-3 Pro w/latest drivers and firmware.

      These servers are connected back-to-back by direct optical link (without switch, to eliminate). I'm using one port. Second port attached to switch and used for management.

      I've used following script for configuration:

       

      #disable everything related to rdma

      Remove-NetQosTrafficClass

      Remove-NetQosPolicy -PolicyStore ActiveStore -Confirm:$False

      Remove-NetQosPolicy -Confirm:$False

      Get-NetAdapterQos  | Disable-NetAdapterQos

      Get-NetAdapterRdma | Set-NetAdapterRdma -Enabled $false

       

      #configure rdma from scratch

      Set-NetQosDcbxSetting -Willing 0

      Set-NetAdapterRdma -Name "CPU2 SLOT6 PCI-E 3.0 X16" -Enabled $true

      New-NetQosPolicy "SMB" -PolicyStore ActiveStore -NetDirectPortMatchCondition 445 -PriorityValue8021Action 4

      New-NetQosPolicy "DEFAULT" -PolicyStore Activestore -Default -PriorityValue8021Action 4

      New-NetQosPolicy "TCP" -PolicyStore ActiveStore -IPProtocolMatchCondition TCP -PriorityValue8021Action 0

      New-NetQosPolicy "UDP" -PolicyStore ActiveStore -IPProtocolMatchCondition UDP -PriorityValue8021Action 0

      Enable-NetQosFlowControl -Priority 4

      Enable-NetAdapterQos -InterfaceAlias "CPU2 SLOT6 PCI-E 3.0 X16"

      New-NetQosTrafficClass -name "SMB class" -priority 4 -bandwidthPercentage 50 -Algorithm ETS

       

      I'm testing RDMA with nd_rping and dumping traffic with ibdump, and that's what I've found.

      Client send frame with "CM: ConnectRequest" command to server. Destination MAC Address is correct in that frame, but source MAC addres is 00:00:00:00:00:00.

      Server receives this frame and relpies with "CM: ConnectReply". In this frame both Source and destination MACs are 00:00:00:00:00:00. Client doesn't receive this frame, so, handshake can't finish.

      This also accompanied by strange frames with "UD Send Only" commands. They are strange because they don't have Vlan and priority tag and part of them are with Source Address that equals Destination Address and another part - with Destination address 0.0.0.0.

      I'm attaching dumps from both client and server.

      Could anyone explain, what's happening?

       

      I also have 4 servers with ConnectX-4 EN adapters, and there are no problems with them - RDMA and SMB Direct works well there.

        • Re: RoCE not working on Win 2016 (ConnectX-3 Pro)
          dkawula

          Hey Dmitry,

           

          I just wrote a blog post about this issue on my site http://www.checkyourlogs.net/?p=36172 today.

           

          Basically without reading too much below --> I just pasted for those other readers looking to do some more research.   We have in the community confirmed a bug with the 2.40.7000 firmware.   I have had several people contact me on twitter today regarding the post and they confirmed the same issue.   Basically we rolled back the firmware and in some cases driver and it fixed this.

           

          One of my readers said he setup a Storage Spaces Direct Configuration for a customer.  He left it for a month and they upgraded the firmware and it all presented the exact same issues below.   They rolled back their firmware and have the same experience as me.

           

          Anyways --> Here is a snip from what I wrote up today.   I haven't heard anything back from mellanox Technical Support yet as I have a request in for someone via the Partner Channel to have them contact me regarding this.   No answers yet.

           

           

          Thanks,

           

          Dave

           

          Here is a snip from my post.

           

           

          Hey Storage Spaces Direct fans I have been working on a deployment for a customer in Washington (Health Care).

          We are building out a development environment for them and they have purchased Mellanox NIC's for both DEV and Production.   (Mellanox CX3-Pro MCX-312B-XCCT)

          Like any normal, deployment we decided to grab the latest firmware and drivers from Mellanox’s website.

          We downloaded the latest Drivers and Firmware and this is what we get for performance on the cards when copying to the CSV Volume in the Cluster.

          10 MB/Sec are you kidding me… (We tried everything before looking at this firmware:  Changed the Avago Controller “LSI 9300”, Updated Firmware on our drives, tore apart the Failover Cluster, tried every conceivable configuration change in Storage Spaces Direct.   Finally, I noticed the Firmware version in my lab and the customer’s was different. So I upgraded mine and this is what happened.

           

          Mellanox1.png

           

          For all you Networking junkies out there… I can repo this on demand every time I upgrade the firmware to 2.40.7000

           

           

          I am able to somewhat fix the issue by reverting the firmware, reinstalling the old driver, and tearing down the entire Virtual networking stack.  

           

          Here are the detailed steps I followed to fix this:

           

          1. Revert the Firmware on the Card to:  2.40.5030 (fw-ConnectXPro-rel-2_40_5030-MCX3112B-XCC_Ax-FlexBoot-3.4.746.bin)
          2. Uninstall the Mellanox Driver (This will break your networking Setup but that is fine)
          3. Reinstall the Mellanox Driver with the older version: 5.35.51100.0 (MLNX_VPI_WinOF-5_35_All_win2016_x64)

           

           

          4. Restart the Server

          5. Validate that the Firmware and Driver are updated in Device MGR

          6. Rebuilt the SET Team / VMSwitch / and Virtual Adapters / Re-Enable RDMA

           

          #Remove Virtual Adapters

          Get-VMNetworkAdapter -ManagementOS SMB_3 | Remove-VMNetworkAdapter
          Get-VMNetworkAdapter -ManagementOS SMB_4 | Remove-VMNetworkAdapter

          Get-VMSwitch -Name teamedvswitch02 | Remove-VMSwitch -confirm:$false

          New-VMSwitch -Name teamedvswitch02 -NetAdapterName "Ethernet 3", "Ethernet 4" -EnableEmbeddedTeaming $true -Confirm:$false

          Add-VMNetworkAdapter -SwitchName teamedvswitch02 -name SMB_3 -ManagementOS
          Add-VMNetworkAdapter -SwitchName teamedvswitch02 -name SMB_4 -ManagementOS
          Enable-NetAdapterRdma "vethernet (SMB_3)","vethernet (SMB_4)"
          Get-NetAdapterRdma

           

           

           

          7. Reconfigure the IP Addresses on the SMB_3 and SMB_4 virtual adapters

           

          This is still slow as it is about 1/2 of the speed I used to get in my lab prior to touching the drivers and firmware.   I was getting 500-700 MB /sec before I touched the drivers and firmware.

           

          I also tested this configuration upgrading to the latest Mellanox Driver –  MLNX_VPI_WinOF-5.35_All_Win2016_x64 (5.35.12978.0 “Device MGR” 5.35.52000.0 “on the Driver Package”.    My results were the same à Better than the 10 MB/sec but not faster than 200 MB /sec yet.   

           

          I did some digging on the Firmware and this is what I have been able to come up with so far:

           

          There only appear to be two fixes in this latest firmware:

          RM#980151: Fixed an issue where a virtual MAC address which is configured by set_port (ifconfig), remained after driver restart

          RM#913926: Fixed an issue where the two interfaces reported the same MAC address when bonding configuration was used.  

           

          http://www.mellanox.com/pdf/firmware/ConnectX3Pro-FW-2_40_7000-release_notes.pdf

           

          I don’t think it will be that big of a deal NOT installing this Firmware and going with the latest drivers.   At least until Mellanox comes back to the community with a fix and or workaround.

           

          There was something rather disturbing that I read in the release notes for this flakey firmware. It appears to me that no testing was done on this firmware one Windows Server 2016.   I may to totally off base here and I am just going off the release notes published on the www.mellanox.com website for this firmware.

           

           

           

           

            • Re: RoCE not working on Win 2016 (ConnectX-3 Pro)
              demonixed

              Hi, Dave!

              Thanks for confirming, I'm not alone with this problem.

               

               

              I'd also like to say, we have faced with this problem while running previous version of firmware (2.40.5032) and drivers (5.35.12970.0). I've saw, that 2.40.7000 fixes two [irrelevant] bugs, but updated this version with hope it will suddenly fix RoCE.

              Actually, I'm not building S2D, but trying to connect Hyper-V hosts to SOFS via RDMA, and see, that RDMA is not working at all (nd_rping, nd_send_bw can't establish RoCE connection).

               

               

              Btw, even without using RDMA we have problems with these servers - virtual machines are constantly lose connection to storage (which is on the SOFS). I think now, in may be caused by firmware too.

               

               

              I'll try to downgrade to older fw/drivers...

                • Re: RoCE not working on Win 2016 (ConnectX-3 Pro)
                  demonixed

                  I've downgraded to oldest driver for windows 2016 (v 5.25, the first firmware for windows 2016) and choosed option flash firmware included with these drivers.

                  RoCE works now!

                  • Re: RoCE not working on Win 2016 (ConnectX-3 Pro)
                    demonixed

                    I have another important addition.

                    Everything was working just after driver&firmware rollback/installation.

                    After that I've added VmSwitch (New-VMSwitch -Name "Embedded Cluster Switch" -NetAdapterName "Port2" -EnableEmbeddedTeaming $true), which was bound just to one adaper (Port 2 on physical nic) (I remind, that second adapter (Port 1 on physical nic) is used for back-to-back connection between servers and I'm using if for testing RDMA) RoCE suddenly stopped working.

                    After several reconfigurations I finally had seen, what everything breaks after I add VmSwitch on the server, which acts as a client in nd_rping, because source MAC address became 00:00:00:00:00:00 after adding VmSwitch.

                     

                    Btw, driver release notes lists something similar to this problem in known issues: 909040 Description: When configuring ConnectX-3 Pro single port via the driver, the RDMA connection fails. WA: N/A Keywords: ConnectX-3 Pro, single port, RDMA.

                    But I can't understand, what is "configuring ConnectX-3 Pro single port via the driver"....

                • Re: RoCE not working on Win 2016 (ConnectX-3 Pro)
                  stepheny

                  https://community.mellanox.com/docs/DOC-1853

                  Note: "Responder CQE Error" counter may raise at an end of SMB direct tear-down session. Details: RDMA receivers need to post receive WQEs to handle incoming messages, if the application does not know how many messages are expected to be received (e.g. by maintaining high level message credits) they may post more receive WQEs than will actually be used. On application tear-down, if the application did not use up all of it’s receive WQEs the device will issue completion with error for these WQEs to indicate HW does not plan to use them, this is done with a clear syndrome indication of “Flushed with error”.

                  1 of 1 people found this helpful
                  • Re: RoCE not working on Win 2016 (ConnectX-3 Pro)
                    bgad15

                    Firmware version 2.42.500 is now out.

                     

                    I can't see any notes regarding this issue in the release notes, either in the fixes or known bugs....

                     

                    Anyone been able to test?