I'm trying to configure SMB Direct on windows 2016 with ConnectX-3 Pro EN card (MCX312C-XCCT), but it's not working. RDMA Activity counters shows constantly rising Responder CQE Errors. So, I treied to buils simplest configuration and now I have the following
Two servers with WIn 2016 w/latest updates. ConnectX-3 Pro w/latest drivers and firmware.
These servers are connected back-to-back by direct optical link (without switch, to eliminate). I'm using one port. Second port attached to switch and used for management.
I've used following script for configuration:
#disable everything related to rdma
Remove-NetQosPolicy -PolicyStore ActiveStore -Confirm:$False
Get-NetAdapterQos | Disable-NetAdapterQos
Get-NetAdapterRdma | Set-NetAdapterRdma -Enabled $false
#configure rdma from scratch
Set-NetQosDcbxSetting -Willing 0
Set-NetAdapterRdma -Name "CPU2 SLOT6 PCI-E 3.0 X16" -Enabled $true
New-NetQosPolicy "SMB" -PolicyStore ActiveStore -NetDirectPortMatchCondition 445 -PriorityValue8021Action 4
New-NetQosPolicy "DEFAULT" -PolicyStore Activestore -Default -PriorityValue8021Action 4
New-NetQosPolicy "TCP" -PolicyStore ActiveStore -IPProtocolMatchCondition TCP -PriorityValue8021Action 0
New-NetQosPolicy "UDP" -PolicyStore ActiveStore -IPProtocolMatchCondition UDP -PriorityValue8021Action 0
Enable-NetQosFlowControl -Priority 4
Enable-NetAdapterQos -InterfaceAlias "CPU2 SLOT6 PCI-E 3.0 X16"
New-NetQosTrafficClass -name "SMB class" -priority 4 -bandwidthPercentage 50 -Algorithm ETS
I'm testing RDMA with nd_rping and dumping traffic with ibdump, and that's what I've found.
Client send frame with "CM: ConnectRequest" command to server. Destination MAC Address is correct in that frame, but source MAC addres is 00:00:00:00:00:00.
Server receives this frame and relpies with "CM: ConnectReply". In this frame both Source and destination MACs are 00:00:00:00:00:00. Client doesn't receive this frame, so, handshake can't finish.
This also accompanied by strange frames with "UD Send Only" commands. They are strange because they don't have Vlan and priority tag and part of them are with Source Address that equals Destination Address and another part - with Destination address 0.0.0.0.
I'm attaching dumps from both client and server.
Could anyone explain, what's happening?
I also have 4 servers with ConnectX-4 EN adapters, and there are no problems with them - RDMA and SMB Direct works well there.