In some cases, the network adapter is connected to the second numa node. While running low latency benchmarks (e.g. osu_latency) on the first numa node we lead to additional latency results.
1. Check with port is being connected to your network
mlx5_0 port 1 ==> ib0 (Up)
mlx5_1 port 1 ==> ib1 (Down)
in this case ib0.
2. Run the following command:
$ cat /sys/class/net/ib0/device/numa_node
Numa 1 is the second numa. normally there are numa 0 and numa 1.
Another possible option to check is via lspci command.
$ lspci | grep Mel
d8:00.0 Infiniband controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
d8:00.1 Infiniband controller: Mellanox Technologies MT28800 Family [ConnectX-5 Ex]
In most cases, the adapters that located on the PCI address that doesn't start with 00 is the connected to the second CPU. In this case it starts with d8.
Now that you know that the adapter is connected to the second CPU, run your benchmark tests aligned.