HowTo Configure PXE Server in UEFI Mode for IPoIB Network

Version 10

    This post shows how to configure PXE server boot on RHEL 7 in UEFI Mode on when running over InfiniBand network.

     

    Important Note: if you wish to run it over Ethernet network, it is similar procedure. the only change is the MAC address for the Ethernet used by the DHCP server.

     

    References

     

    Setup

    The setup that requires here is basically one PXE server and one Host that will be the PXE client.

    The network should be running InfiniBand.

     

    The Services required on the PXE Server:

    1. TFTP

    2. DHCP

    3. NFS

    1%3Fauth_token%3De054014612de078648604902cd12b40700d3645c

     

    Prerequisites

    1. Install RHEL7 on the PXE server plus latest MLNX_OFED

    2. InfiniBand network is up (SM is enabled, ports are up ...)

     

    PXE Server Configuration

     

    Setting up the InfiniBand interface (ib0)

    1. Make sure that the post is set to work in InfiniBand mode.

    Refer to HowTo Change Port Type in Mellanox ConnectX-3 Adapter

     

    2. Get MAC Address for InfiniBand Port. Type “ifconfig” and copy the first (12) hexadecimals.

    # ifconfig ib0
    ...

    ib0     Link encap: Infiniband HWaddr A0:00:02:20:FE:80:00:00:00:00:00:00:00:00:00:00:00:00:00:0

            inet 10.5.5.5  Bcast:netmask 10.5.5.255 Mask:255.255.255.0

            BROADCAST MULTICAST MTU:2044 Metric:1

            RX packets:76077 errors:0 dropped:0 overruns:0 frame:0

            TX packets:76354 errors:0 dropped:0 overruns:0 carrier:0 

            collisions:0 txqueuelen:1024

            RX BYTES:2445544 (2.3 MiB) TX bytes:91270253 (87.0 MiB)

     

    3. Type “ibstat” and copy the Port “GUID” address

    # ibstat

    ...

    Port 1:

                    State: Active

                    Physical state: LinkUp

                    Rate: 40

                    Base lid: 2

                    LMC: 0

                    SM lid: 2

                    Capability mask: 0x0251486a

                    Port GUID: 0x24be05ffffac8902

                    Link layer: InfiniBand

    Combine the two addresses to get the Hardware address of the InfiniBand port (to be used later on the ifcfg-ib0 file):

    A0:00:0A:20:FE:00:00:00:00:00:00:24:BE:05:FF:FF:AC:89:02

     

    4. Create ifcfg-ib0 file.

    Make sure to add the hardware address to the HWADDR field:

    # cat /etc/sysconfig/network-scripts/ifcfg-ib0

    ...

    DEVICE=ib0

    HWADDR=A0:00:0A:20:FE:00:00:00:00:00:00:24:BE:05:FF:FF:AC:89:02

    TYPE=Infiniband

    IPADDR=10.5.5.5

    NETMASK=255.255.255.0

    ONBOOT="yes"

    NM_CONTROLLED="yes"

     

    5. Enable the InfiniBand interface

    # ifup ib0

     

    Configure TFTP Server

    1. Install relevant packages

    # yum install tftp xinetd tftp-server


    2. Create a “uefi” directory in your TFTP directory

    # mkdir –p /var/lib/tftpboot/uefi


    3. Verfy that the TFTP config file matches the below settings and it is enabled (disable = no).

    # cat /etc/xinetd.d/tftp

    ...

    service tftp

    {

    protocol        = udp

    socket_type     = dgram

    wait            = yes

    user            = root

    server          = /usr/sbin/in.tftpd

    server_args     = -s /var/lib/tftpboot

    disable         = no

    per_source      = 11

    cps             = 100 2

    flags           = IPv4

    }


    4. Restart TFTP

    # /etc/init.d/xinetd restart

     

    5. Enable TFTP on boot

    # /sbin/chkconfig tftp on

     

     

    Configure DHCP Server

    1. Install DHCP

    # yum install dhcp

     

    2. Make the InfiniBand port (e.g. ib0) the primary DHCP port.

    # vi /etc/sysconfig/dhcpd

    ...

    DHCPDARGS=ib0

     

    3. Edit the DHCP configuration file to look like below example.

    In this example, the subnet 10.5.5.0 is used for the PXE client. it is also possible to assign static IP address per server (Need to use the hardware address field).

    Refer to the DHCP manual as well (click here).

    # vi /etc/dhcp/dhcpd.conf
    ...
    #

    # DHCP Server Configuration file.
    #   see /usr/share/doc/dhcp*/dhcpd.conf.sample
    #   see 'man 5 dhcpd.conf'
    #
    default-lease-time                      28800;
    max-lease-time                          28800;
    option subnet-mask                      255.255.255.0;
    option domain-name                      "drbl.org";
    ddns-update-style                       interim;
    server-name                             drbl;
    ignore client-updates;
    option client-architecture code 93 = unsigned integer 16;
    option broadcast-address 255.255.255.255;
    allow booting;

    option space PXE;
      option PXE.mtftp-ip    code 1 = ip-address;
      option PXE.mtftp-cport code 2 = unsigned integer 16;
      option PXE.mtftp-sport code 3 = unsigned integer 16;
      option PXE.mtftp-tmout code 4 = unsigned integer 8;
      option PXE.mtftp-delay code 5 = unsigned integer 8;
      option arch code 93 = unsigned integer 16; # RFC4578

     

    subnet 10.5.5.0 netmask 255.255.255.0 {
        option subnet-mask  255.255.255.0;
        option routers 10.5.5.5;
        next-server 10.5.5.5;
        always-broadcast on;
        filename "uefi/shim.efi";

        pool {

              range 10.5.5.20 10.5.5.30;
            }

    }

     

    4. Restart the DHCP server.

    # /etc/init.d/dhcpd restart

     

    Configure NFS Server

    1. Make sure that NFS is running

    # /etc/init.d/nfs restart

     

    2. Make the InfiniBand port (e.g. ib0) the primary DHCP port.

    # vi /etc/sysconfig/dhcpd

    ...

    DHCPDARGS=ib0

     

    3. Your mount point will act as a repository.

    To add the mount point add the below command to the ‘/etc/exports’ file, restart NFS, and run ‘exportfs’.

    The repository should show <world> which means that it is open to all servers.

    # echo /home/sw/os/mp *\(rw,sync,no_root_squash\)>> /etc/exports
    # /etc/init.d/nfs restart

    # exportfs | grep mp

    ...

    /home/sw/os/mp  <world>

     

    Create repository for UEFI PXE Booting

    1. Create a repository by copying RHEL 7 ISO file to a directory (ex. /home/sw), and mount the ISO to the NFS mount point.

    Note: It is recommended to create the ISO image from the OS and the MLNX_OFED driver as well.

    The mount is done in order to open the ISO image to directories.

    # mkdir –p /home/sw/os/mp

    # scp 10.X.X.X:/RHEL-7.0-Server*.iso /home/sw/os/

    # mount –o loop /home/sw/RHEL-7.0-Server*.iso /home/sw/os/mp

    Copy the PXE files from RHEL 7 ISO

    1. Create a rpm directory (ex. /home/sw/os/rpm_rhel7) and copy the ‘shim’ and ‘grub2-efi’ rpms from the "Packages" folder located within the mount point.

    # mkdir –p /home/sw/os/rpm_rhel7

    # cd /home/sw/os

    # cp –pr mp/Packages/shim-*.rpm rpm_rhel7/

    # cp –pr mp/Packages/grub2-efi-*.rpm rpm_rhel7/

    # cd rpm_rhel7

    # ls

    ...

    grub2-efi-2.02-0.2.10.el7.x86_64.rpm  shim-0.7-5.2.el7.x86_64.rpm


    2. Convert the rpm packages to archives.

    Note: The ‘shim’ package will create a ‘boot’ directory and the ‘grub2-efi’ package will create an ‘etc’ and  'usr’ directory.

    # rpm2cpio shim-0.7-5.2.el7.x86_64.rpm | cpio –dimv

    # rpm2cpio grub2-efi-2.02-0.2.10.el7.x86_64.rpm | cpio -dimv

    # ls

    ...

    boot  grub2-efi-2.02-0.2.10.el7.x86_64.rpm  shim-0.7-5.2.el7.x86_64.rpm  usr

    etc

     

    3. Copy the ‘shim.efi’ and ‘grubx64.efi’ files, as well as the ‘grub.cfg’, ‘vmlinuz’, and ‘initrd.img’ files to your TFTP directory.

    # cp /home/sw/rpm/boot/efi/EFI/redhat/shim.efi /var/lib/tftpboot/uefi/

    # cp /home/sw/rpm/boot/efi/EFI/redhat/grubx64.efi /var/lib/tftpboot/uefi/

    # cp /home/sw/mp/EFI/BOOT/grub.cfg /var/lib/tftpboot/uefi/

     

    # mkdir –p /var/lib/tftpboot/uefi/images/pxeboot

    # cp /home/sw/mp/images/pxeboot/vmlinuz /var/lib/tftpboot/uefi/images/pxeboot/

    # cp /home/sw/mp/images/pxeboot/initrd.img /var/lib/tftpboot/uefi/images/pxeboot/

     

    4. Edit the ‘grub.cfg’ file to point to the folder carrying the boot files and to include a boot repository.

    The below example uses NFS.

    Note: without the “inst.” prefix RHEL7 will not boot properly in UEFI mode.

    Refer to https://docs.oracle.com/cd/E52668_01/E54695/html/ol7-install-boot-options.html for more boot options.

    vi /var/lib/tftpboot/uefi/grub.cfg

    ...

    set timeout=60
    ### END /etc/grub.d/00_header ###

    search --no-floppy --set=root -l 'MENU LIST'

    ### BEGIN /etc/grub.d/10_linux ###
    menuentry 'Install Red Hat Enterprise Linux 7.0' --class fedora --class gnu-linux --class gnu --class os {
            linuxefi uefi/images/pxeboot/vmlinuz ro devfs=nomount inst.repo=nfs:10.5.5.5:/home/sw/os/mp/
            initrdefi uefi/images/pxeboot/initrd.img

    }

    Host (PXE Client) BIOS Configuration

    1. In the BIOS, set client to boot in UEFI mode and reboot client server.

     

    This example is from HP ProLiant DL380 Gen9 ( HP BIOS Version P89  Release Date: 01/28/2015, Revision 1.40, Firmware 2.30)

     

     

    2. During boot, Press F12 and the client server should PXE boot RHEL 7 in UEFI mode.