Latest CentOS Updates Cause Serious Boot Issues on Hyper-V

On or about September 18th 2018, CentOS rolled out an update which included an updated Linux kernel, various system-wide patches and an updated version of the packages “shim-x64” and “mokutil”. Changes to these packages resulted in default files critical to system boot configuration being modified, as well as their default locations being changed.

Part of the update was intended to fix a long-pending bug related to EFI, Secure Boot and Hyper-V with Linux Virtual Machines. This issue has since been addressed via a work-around using a bootstrapping method within Hyper-V. Unfortunately, this bootstrapping method is effectively broken by the update.

Problem
When using Generation 2 Virtual Machines, Hyper-V manages a copy of “shimx64.efi” external to the system disk which means boot sector changes on-disk are not carried over outside the VM to the Hyper-V system.

As a result of this update some CentOS-based Virtual Machine systems running in Microsoft Hyper-V environments are not able to successfully boot post-update due to Hyper-V being unaware of the changes to the boot sector and the location of critical boot files being in a different location than Hyper-V expects.
Rebooting a system after updating any or all of the following may result in Hyper-V reporting it cannot find the EFI system files, making the Virtual Machine unbootable:

• Linux Kernel version 3.10.0-862.14.4.el7.x86_64
• shim-x64 version 12.2.el7
• mokutil version 12.2.el7

On Virtual Machines creating using the Generation 2 configuration updating any one of these items could result in the boot issue, as they are all dependencies of one another and all would be updated together.

Solution
In order to resolve this issue, the missing boot configuration files can be copied into the location where Hyper-V expects them to be so that Hyper-V is now able to detect the proper EFI configuration. It is important that this is done after updating and before rebooting to avoid requiring a system administrator to resolve this in Hyper-V manager.

It is also important to note that while the VM will be bootable at this point, it will still be booting an older kernel. This can be changed by removing old kernel versions from the boot directory and rebuilding the GRUB bootloader configuration. This will require an additional reboot to take effect.

This effectively resolves the EFI boot issue going forward. Linux will now be updated to the latest kernel and should boot normally during subsequent boot cycles. to I updated a bit of the wording on this to be a bit more accurate that it doesn’t happen to all machines and to clearly explain that it’s post-update.

Detailed Instructions
Overview
Post-Update, Pre-Reboot Fix
Post-Update, Post-Reboot Fix