0

I have an Ubuntu machine that has been through several kernel upgrades. At the start of the day, I had 3 kernels installed: 5.11.0-34, 5.11.0-46, and 5.11.0-49. I had to upgrade a bunch of packages, and afterward took the opportunity to remove the middle kernel to open up room in my boot partition.

Now, I cannot get either remaining kernel to boot. Neither of them prompts for the password to decrypt the drive where Linux is installed. It doesn't matter whether I boot into recovery mode or not, they print messages and eventually drop into a shell like this:

Unable to init MCE device (rc: -5)
Volume group "vgubuntu" not found
Cannot process volume group vgubuntu
Gave up waiting for suspend/resume device
Gave up waiting for root file system device. Common problems:
 - Boot args (cat /proc/cmdline)
   - Check rootdelay= (did the system wait long enough?)
 - Missing modules (cat /proc/modules; ls /dev)
ALERT!  /dev/mapper/vgubuntu-root does not exist.  Dropping to a shell!

BusyBox v1.30.1 (Ubuntu 1:1.30.1-6ubuntu2.1) built-in shell (ash)

I have found a similar account here, but I don't have any devices in /dev with a name prefixed with sda. I'm guessing that's because I don't have any SATA drives. My drives are all NVMe, and I don't know which one has the root file system.

Long ago I added mce=off as a kernel parameter. It is present in every GRUB menu option.

How can I fix my installation to boot?

John Freeman
  • 101

1 Answers1

0

Something got borked somewhere and I had to run update-initramfs. I found very similar instructions in three separate places:

  1. https://ubuntuforums.org/showthread.php?t=2409754&s=e1f324bf5e566b3bb93374cd07bdcc17&p=13828993
  2. https://askubuntu.com/a/868726/538768
  3. https://feeding.cloud.geek.nz/posts/recovering-from-unbootable-ubuntu-encrypted-lvm-root-partition/

Here's how I got there.

I loaded Ubuntu off a live USB and ran fdisk -l to see my partitions and guess which one was encrypted. I saw these (among others):

  • /dev/nvme2n1p1: 512M EFI System
  • /dev/nvme2n1p2: 732M Linux filesystem
  • /dev/nvme2n1p3: 1.8T Linux filesystem <-- I guessed it was this one.

Then I decrypted the partition and mounted it like this:

sudo -i
cryptsetup open /dev/nvme2n1p3 $name
vgchange -ay
mkdir /mnt/root
mount /dev/mapper/$name /mnt/root

That let me inspect /etc/crypttab to see which device name to use when decrypting the partition (nvme0n1p3_crypt in this case):

nvme0n1p3_crypt UUID=743ab129-75bb-429b-8366-9c066f00c4fe none luks,discard

Then I looked at /etc/fstab to see which partitions were the boot partition and EFI partition:

# /boot was on /dev/nvme0n1p2 during installation
UUID=773ceeb2-5c0f-4838-baad-a1182d7fdd80 /boot           ext4    defaults        0       2
# /boot/efi was on /dev/nvme0n1p1 during installation
UUID=5C17-FB32  /boot/efi       vfat    umask=0077      0       1

At installation, these partitions were named like nvme0n1p*, but no longer. I could find their current names by listing /dev/disk/by-uuid:

$ ls -l /dev/disk/by-uuid/
lrwxrwxrwx 1 root root 15 Jan 31 12:29 5C17-FB32 -> ../../nvme2n1p1
lrwxrwxrwx 1 root root 15 Jan 31 12:29 743ab129-75bb-429b-8366-9c066f00c4fe -> ../../nvme2n1p3
lrwxrwxrwx 1 root root 15 Jan 31 12:29 773ceeb2-5c0f-4838-baad-a1182d7fdd80 -> ../../nvme2n1p2

Now I had all the pieces I needed to follow the instructions. Here are the actual commands I executed:

sudo -i
cryptsetup open /dev/nvme2n1p3 nvme0n1p3_crypt
mount /dev/mapper/nvme0n1p3_crypt /mnt/root
mount /dev/nvme2n1p2 /mnt/root/boot
mount /dev/nvme2n1p1 /mnt/root/boot/efi
mount --bind /dev /mnt/root/dev
mount --bind /run /mnt/root/run
chroot /mnt/root
mount -t proc proc /proc
mount -t sysfs sys /sys
update-initramfs -c -k all

Then I was able to restart the machine and boot into one of the installed kernels.

John Freeman
  • 101