This post discusses the iommu and intel_iommu Linux grub parameters for SR-IOV pass-through (pt) mode. When working in an SR-IOV environment, we need to make sure that kernel enables SR-IOV and that we get good performance.
To enable SR-IOV in the kernel, configure intel_iommu=on in the grub file. To get the best performance, add iommu=pt (pass-through) to the grub file when using SR-IOV. When in pass-through mode, the adapter does not need to use DMA translation to the memory, and this improves the performance. iommu=pt is needed mainly with hypervisor performance is needed.
Setup and Prerequisites
One Linux server with Virtualization environment. You can follow one of the posts below per adapter type and technology used (Ethernet, InfiniBand).
- HowTo Configure SR-IOV for ConnectX-3 with KVM (InfiniBand)
- HowTo Configure SR-IOV for ConnectX-3 with KVM (Ethernet)
- HowTo Configure SR-IOV for Connect-IB/ConnectX-4 with KVM (InfiniBand)
- HowTo Configure SR-IOV for ConnectX-4 with KVM (Ethernet)
grub file configuration
Depending on the Linux distribution, you may use grub or grub2.
grub files are located by default at the following paths:
grub file example:
Mellanox Technologies Multiboot GRUB Configuration
#Current Os number: 01
kernel /vmlinuz-Ub15.10x86_64-4.2.0-42-generic root=/dev/sda2 console=tty0 console=ttyS0,115200n8 rhgb intel_iommu=on iommu=pt
1. After adding the parameters to the grub file, reboot the server.
2. After reboot, check dmesg and check that your PCI device isn't in the list of RMRR setting , if you still see it, you need to contact your server vendor(we saw the issue on HP Gen8).
[ 4.027939] DMAR: Setting RMRR:
[ 4.089276] DMAR: Setting identity map for device 0000:02:00.0 [0xe8000 - 0xe8fff]
[ 4.097709] DMAR: Setting identity map for device 0000:03:00.0 [0xe8000 - 0xe8fff]
Refer to https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt to learn more about DMAR and RMRR functions.
3. Use the lspci command to confirm you have the list of PCI devices (VFs) as configured.