4 Replies Latest reply on Jul 16, 2018 7:40 AM by andrzejp

    Windows RDMA QoS and WinOF-2 1.80 issues



      so I'm setting up Scale-Out file server on Windows Server 2016 (fully patched for this date). When I use WinOF-2 newer then 1.70 (tried 1.80.50 and 1.80.51) The QoS is not working properly.

      I'm using MXC416A-BCAT with firmware 12.21.2010.

      I get RDMA to work but "WinOF-2 Port QoS" indicate that incorrect priorities are used and traffic is marked with wrong DSCP values. Also Get-NetAdapterQos indicates some kind of a problem.


      My configuration:

      Set-NetAdapterAdvancedProperty -Name SMB1 -RegistryKeyword "*JumboPacket" -RegistryValue 4200
      Set-NetAdapterAdvancedProperty -Name SMB1 -RegistryKeyword VlanID -RegistryValue "4029"  
      Set-NetQosDcbxSetting -Willing 0 -Confirm:$false
      Enable-NetQosFlowControl  -Priority 4,5
      Disable-NetQosFlowControl  -Priority 0,1,2,3,6,7
      New-NetQosTrafficClass "SMB" -Priority 5 -BandwidthPercentage 95 -Algorithm ETS
      New-NetQosPolicy "SMBD" -NetDirectPortMatchCondition 445 -PriorityValue8021Action 5 -Confirm:$false -Precedence 255 -NetworkProfile All
      New-NetQosPolicy "SMB" -SMB -PriorityValue8021Action 5 -DSCPAction 42 -Confirm:$false -Precedence 255 -NetworkProfile All
      New-NetQosPolicy "DEFAULT" -Default -PriorityValue8021Action 5 -DSCPAction 42 -Confirm:$false -Precedence 240 -NetworkProfile All
      New-NetQosPolicy "CLUSTER" -Cluster -PriorityValue8021Action 7 -DSCPAction 56 -Confirm:$false -Precedence 240 -NetworkProfile All
      New-NetQosPolicy "TCP"  -IPProtocolMatchCondition TCP -PriorityValue8021Action 0 -DSCPAction 0 -Confirm:$false -Precedence 10 -NetworkProfile All
      New-NetQosPolicy "UDP"  -IPProtocolMatchCondition UDP -PriorityValue8021Action 0 -DSCPAction 0 -Confirm:$false -Precedence 10 -NetworkProfile All
      Enable-NetAdapterQos -Name  SMB1
      Set-NetAdapterQos -Name  SMB1 -Enabled $true
      new-itemProperty -Path $m -Name "RxTrustedState" -PropertyType "String" -Value "2" -Force
      new-itemProperty -Path $m -Name "TxUntagPriorityTag" -PropertyType "String" -Value "1" -Force
      new-itemProperty -Path $m -Name "PriorityToDscpMappingTable_7" -PropertyType "String" -Value "56" -Force 
      new-itemProperty -Path $m -Name "PriorityToDscpMappingTable_5" -PropertyType "String" -Value "42" -Force 
      new-itemProperty -Path $m -Name "PriorityToDscpMappingTable_4" -PropertyType "String" -Value "34" -Force 
      Mlx5Cmd.exe -Qosconfig -Dcqcn -Name SMB1 -Set -DcqcnCnpDscp 58 -DcqcnCnp802pPrio 7
      Mlx5Cmd.exe -Qosconfig -Dcqcn -Name SMB1 -Disable
      Mlx5Cmd.exe -Qosconfig -Dcqcn -Name SMB1 -Enable 4
      Mlx5Cmd.exe -Qosconfig -Dcqcn -Name SMB1 -Enable 5
      #Mlx5Cmd.exe -Qosconfig -Dcqcn -Name "SMB1" -Set –DcqcnCnpPrioMode 0 #btw. this is not working - Invalid command or flag,
      Restart-NetAdapter SMB
      Enable-NetAdapterRDMA SMB1



      So I want to see SMB Direct on priority 5 with DSCP 42. However counters indicate priority 3 is used instead of 5.

      So I'm checking:

      PS C:\Windows\system32> Get-NetAdapterqos SMB1
      Name         : SMB1
      Enabled      : True
      Capabilities :                       Hardware     Current     
                                           --------     -------     
                     MacSecBypass        : NotSupported NotSupported
                     DcbxSupport         : IEEE         IEEE        
                     NumTCs(Max/ETS/PFC) : 1/1/1        1/1/1       

      This is clearly wrong.

      Then I reboot the server. And get the same result.

      Then I check QoSPolicy:

      PS C:\Windows\system32> Get-NetQosPolicy -PolicyStore "ActiveStore" | select elementname,name,owner,NetworkProfile,Precedence,IPProtocol,Template,PriorityValue,DSCPValue,DSCPAction,NetDirectPort | sort -Descending Precedence,elelementname | ft
      elementname   name          owner                  NetworkProfile Precedence IPProtocol      Template PriorityValue DSCPValue DSCPAction
      -----------   ----          -----                  -------------- ---------- ----------      -------- ------------- --------- ----------
      livemigration livemigration Group Policy (Machine)            All        255       None LiveMigration             4        34         34
      smb           smb           Group Policy (Machine)            All        255       None           SMB             5        42         42
      smbd          smbd          Group Policy (Machine)            All        255       None          None             5        -1         -1
      cluster       cluster       Group Policy (Machine)            All        240       None       Cluster             7        56         56
      default       default       Group Policy (Machine)            All        240       None       Default             5        42         42
      tcp           tcp           Group Policy (Machine)            All         10        TCP          None             0         0          0
      udp           udp           Group Policy (Machine)            All         10        UDP          None             0         0          0


      So policy is ok. But traffic is using priority 3 and DSCP 26 - and these values are not configured anywhere!

      When I use driver 1.70 priorites are used correctly.


      Any one has any ideas what can I do to get priorities working correctly with current driver version ?

        • Re: Windows RDMA QoS and WinOF-2 1.80 issues


          Look at the User Manual for WinOF-2, page 69. You also need to configure some additional registry keys:


          TxUntagPriorityTag If 0x1, do not add 802.1Q tag to transmitted packets which are assigned 802.1p priority, but are not assigned a non-zero VLAN ID (i.e. priority-tagged). Default 0x0, for DSCP based PFC set to 0x1. Note: These packets will count on the original priority, even if the registry is on.

          RxUntaggedMapToLossless If 0x1, all untagged traffic is mapped to the lossless receive queue. Default 0x0, for DSCP based PFC set to 0x1.

          PriorityToDscpMappingTable_<ID> A value to mark DSCP for RoCE packets assigned to CoS=ID, when priority flow control is enabled. The valid values range is from 0 to 63, Default is ID value, e.g. PriorityToDscpMappingTable_3 is 3. ID values range from 0 to 7.

          DscpBasedEtsEnabled If 0x1 - all DSCP based ETS feature is enabled, if 0x0 - disabled. Default 0x0.

          DscpForGlobalFlowControl Default DSCP value for flow control. Default 0x1a.

          RxTrustedState Default using host priority (PCP) is 1 Default using DSCP value is 2

            • Re: Windows RDMA QoS and WinOF-2 1.80 issues



              1. I do not have such issues with 1.70..., haven't tried 1.90 yet, I will in a month.

              2. TxUntagPriorityTag - line 20 of my output.

              3. RxUntaggedMapToLossless - I'm using only vlan tagged traffic so no need to set it in my case.

              4. PriorityToDscpMappingTable - lines 21-23.

              5. DscpBasedEtsEnabled - I missed that one. However in 1.70 it works fine without it set.

              6. DscpForGlobalFlowControl - missed that also, but that is DSCP only for flow control, not for entire traffic.

              7. RxTrustedState - line 19