I've been trying out setting up of partitions in our IB fabric. I'm using a SX6036 with the onboard SM enabled. I have thre nodes which are connected to this switch. I've done the following changes to the partition config.
Default Partition -> FULL Membership for SELF
LIMITED membership for ALL
Test Partition 0x0010 -> Full Membership for GUID's of two of the nodes.
When I try to run ib_send_lat or any of the test programs, I keep getting the following error on the sending side
Completion with error at client
Failed status 12: wr_id 0 syndrom 0x81
In the SM logs, I keep getting a message
log_trap_info: Received Generic Notice type:2 num:257 (Bad P_Key) Producer:1 (Channel Adapter) from ......
What am I doing wrong ?I must mention, when I give FULL Membership to ALL in the Default partition, all nodes can talk to each other, but that's not what I want to achieve.
Any tips are appreciated.
In order to let to the nodes communicate over configured partition child interface should be created for IPoIB interface, or application should specify the P_Key.
Did you try to use ib_send_lat (or other) tools with --pkey_index argument?