The difference between upstream, Mellanox OFED and inbox RHEL drivers is major :
Why is Mellanox OFED distribution needed at all? For example, take RHEL7.3, they essentially backported 4.4.x-4.5.x kernel stuff and they keep backporting fixes & features from upstream stable kernels, so what's the point?
>>The point is that Mellanox cannot guarantee that most of the latest code that deals with functionality, bug fixes and performance optimizations (not necessarily in scripts that are added on top) related ConnectX-3/4/5 drivers, including ULPs, is included in each OS and kernel. The drivers constantly updating, and submitting and qualifying patches to upstream kernel revolves around a heavy process before it any commit actually gets accepted. a commit first gets to be accepted to upstream kernel and then cherry picked to a specific distro kernel. Mellanox OFED provided the options to make sure you are up to date regardless if your use case is IB, Ethernet or just an application using libibverbs.
I do understand it's needed for very old kernels & userspace, where upstream is gone EOL long ago and nobody cares about that any longer. But what about up to date distros, like Ubuntu 16.04, I see your OFED is actually older (looking at rdma headers) than inhouse stuff
>>That is not always the case. Actually, most of the packages in Mellanox OFED are quite different that the ones that come with RHEL 7.3 inbox driver. Take a look at IPoIB, libmlx5, and others.
Would be wonderful to have a KB article explaining why I should install this thing (quite a lot of packages and kernel modules, btw) vs. just going with inhouse/inbox stuff.
>>You don't "have" to install the entire package, For Ethernet you can just use the Mellanox ethernet driver, with an extension of libibverbs if you use RDMA based applications. You can also use Mellanox OFED to specifically cherry-pick packages you need, or alternatavley create your own repository using RPMs that come shipped with it. The point is that you get presented with a number of options, and you get to choose. The recommended version is whatever proves to work with your environment in the most stable way.