Understanding the iommu Linux grub File Configuration

Version 7

    This post discusses the iommu and intel_iommu Linux grub parameters for SR-IOV pass-through (pt) mode. When working in an SR-IOV environment, we need to make sure that kernel enables SR-IOV and that we get good performance.

    To enable SR-IOV in the kernel, configure intel_iommu=on in the grub file. To get the best performance, add iommu=pt (pass-through) to the grub file when using SR-IOV. When in pass-through mode, the adapter does not need to use DMA translation to the memory, and this improves the performance. iommu=pt is needed mainly with hypervisor performance is needed.

     

    References

     

    Setup and Prerequisites

    One Linux server with Virtualization environment. You can follow one of the posts below per adapter type and technology used (Ethernet, InfiniBand).

     

    grub file configuration

    Depending on the Linux distribution, you may use grub or grub2.

    grub files are located by default at the following paths:

    • /boot/grub/grub.conf
    • /boot/grub2/grub.cfg

     

    grub file example:

    Mellanox Technologies Multiboot GRUB Configuration

    default 0

    timeout 10

    #Current Os number: 01

    title Ub15.10x86_64-4.2.0-42-generic

    root (hd0,0)

    kernel /vmlinuz-Ub15.10x86_64-4.2.0-42-generic root=/dev/sda2 console=tty0 console=ttyS0,115200n8 rhgb intel_iommu=on iommu=pt

    initrd /initrd.img-Ub15.10x86_64-4.2.0-42-generic

     

    Verification

    1. After adding the parameters to the grub file, reboot the server.

     

    2. After reboot, check dmesg and check that your PCI device isn't in the list of RMRR setting , if you still see it, you need to contact your server vendor(we saw the issue on HP Gen8).

    [ 4.027939] DMAR: Setting RMRR:

    [ 4.089276] DMAR: Setting identity map for device 0000:02:00.0 [0xe8000 - 0xe8fff]

    [ 4.097709] DMAR: Setting identity map for device 0000:03:00.0 [0xe8000 - 0xe8fff]

    Refer to https://www.kernel.org/doc/Documentation/Intel-IOMMU.txt  to learn more about DMAR and RMRR functions.

     

    3. Use the lspci command to confirm you have the list of PCI devices (VFs) as configured.