My lab is running:
I followed the instructions here: https://forums.servethehome.com/index.php?threads/10gb-sfp-single-port-cheaper-than-dirt.6893/page-9#post-145423
I am able to get IPoIB working and all of the hosts can ping each other.
However, multicast is not working. How can I get multicast to work? It is preventing vSAN from working.
The solution was to upgrade to ESXi 6.5 update 1 which contains vSAN 6.6. vSAN 6.6 removes the need for multicast.