HowTo Compile Linux Kernel for NVMe over Fabrics

Version 14

    This post shows an example to compile new Linux kernel used for NVME over Fabrics. The reader should have some experience with Linux kernel compilation.

    This post was updated for kernel 4.8.

     

    References

     

    Setup

    • Use kernel 4.8 or later. In this example kernel 4.8.7 was used over RHEL 7.2 (It is recommended to use one of the latest distributions).
    • Note: MLNX_OFED (3.4.2 or later) can be installed after this procedure using --add-kernel-support flag. see HowTo Install MLNX_OFED Driver .

     

    Configuration

    NVMe over Fabrics is available on a git repository: nvme-fabrics.git. However, there is no need to use this GIT anymore (unless you are developer). Kernel 4.8 or later includes the modules required.

     

    1. Get the latest stable kernel from The Linux Kernel Archives

    # mkdir /root/4.8.7

    # cd /root/4.8.7

    # wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.8.7.tar.xz .

    ...

     

    2. Open the file:

    # unxz linux-4.8.7.tar.xz

    # tar xf linux-4.8.7.tar

    # cd linux-4.8.7

     

    3. Create the kernel .config file. There are several ways to do so, refer to KernelBuild - Linux Kernel Newbies  for more options.

    In this example I used the current config file under /boot.

    # cp /boot/config-4.7.4-1.el7.elrepo.x86_64 .config

     

    4. Run "make menuconfig" to open the kernel GUI configuration.

     

    Note: you many need to run the following installations before running "make manuconfig", as it requires the following packages.

    # yum install ncurses ncurses-devel openssl-devel grub2

     

    Note: For Ubuntu, you will need the following packages:

    # apt-get install libncurses5-dev libncursesw5-dev

     

    Run:

    # make menuconfig

     

    a. Go to : Device Drivers

     

    b. Select the following modules by clicking on each line and typing the letter "M".

    • NVM Express block device
    • NVM Express over Fabrics RDMA host driver
    • NVMe Target support
    • NVMe loopback device support
    • NVMe over Fabrics RDMA target support

     

     

    c. Verify that Mellanox drivers are enabled.

     

    Go to:

    • Device Drivers   > Network device support > Ethernet driver support > Mellanox devices >

    And make sure that the following module is enabled.

    Mellanox Technologies ConnectX-4 and Connect-IB core driver                                                                                                     

     

     

    d. Click on "Save" and exit the GUI.

     

    5. Make sure that the NVME parameters are as follows in .config file

    # grep "NVME_" .config

     

    CONFIG_NVME_CORE=m

    # CONFIG_BLK_DEV_NVME_SCSI is not set    <-- Ignore this line.

    CONFIG_NVME_FABRICS=m

    CONFIG_NVME_RDMA=m

    CONFIG_NVME_TARGET=m

    CONFIG_NVME_TARGET_LOOP=m

    CONFIG_NVME_TARGET_RDMA=m

     

    6. Compile the kernel and modules. Use lscpu command to file the number of cores and than use "-j" and the number of CPUs to compile the kernel on all cores.

    In this example the numbers of cores is 24.

    # make -j24

    ...

     

    # make -j24 modules_install install

    ...

     

    7. Get the installed kernel names, locate the new installed kernel.

    In this case we are using grub2, which is common on the new OS distributions. Older distributions may use /boot/grub/grub.cfg file.

    Note: the first line is the new kernel that installed on this server.

    # grep ^menuentry /boot/grub2/grub.cfg | cut -d "'" -f2

    CentOS Linux (4.8.7) 7 (Core)

    CentOS Linux (4.7.3) 7 (Core)

    CentOS Linux (4.7.4-1.el7.elrepo.x86_64) 7 (Core)

    CentOS Linux (3.10.0-327.el7.x86_64) 7 (Core)

    CentOS Linux (0-rescue-5ae3bccd0a7c43e7a39ee879ca1efce3) 7 (Core)

     

    Note: the grub file may be located in different path, for example grep ^menuentry /boot/grub2/grub.cfg | cut -d "'" -f2 will work only for non UEFI servers with BIOS only.

    For UEFI Redhat servers grub.cfg file located at  /boot/efi/EFI/redhat/grub.cfg

    So the command for UEFI machine must be:

    # grep ^menuentry /boot/efi/EFI/redhat/grub.cfg | cut -d "'" -f2
    ...

     

    8 Set the new kernel as default.

    # grub2-set-default 'CentOS Linux (4.8.7) 7 (Core)'

     

    9. Reboot the servers with the new kernel.

    # reboot

     

    10. Verify that the new kernel was loaded.

    # uname -r

    4.8.7

    11. MLNX_OFED (3.4.2 or later) can be installed after this procedure using --add-kernel-support flag.

    see HowTo Install MLNX_OFED Driver .