NEW INFO 6/13/24 16:18 MT: after main post
Having problems with apt upgrade on my main workstation, been troubleshooting for months and I can't figure this out. Kernel will not update. Need the machine, so I keep restoring from backups when my troubleshooting invariably borks the machine, but I need help figuring this out. I know this is a long post, but please read through and see if you have any suggestions. I don't want to admit defeat and do a clean install, but I'm at a loss. Not ready to take the plunge for 24.04 yet.
Running Ubuntu 22.04.04 LTS on a Ryzen 5700X w/64G RAM. A few months ago I started getting "update-initramfs: Generating /boot/initrd.img-6.5.0-26-generic; cpio: write error: No space left on device; E: mkinitramfs failure cpio 2" when I would try to do an apt upgrade. I also see that initramfs-tools post installation script returns exit code of 1. Normally I'd expect I was running out of space in /boot, however on my system /boot is part of /, and has about 44G free. /boot/efi is its own 512M fat32 filesystem with 505M free. In fact, nothing is full, as you can see below:
root@jaeger:~# df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 6.3G 2.7M 6.3G 1% /run
/dev/mapper/vgubuntu-lv_root 49G 3.0G 44G 7% /
/dev/mapper/vgubuntu-lv_usr 20G 7.9G 11G 43% /usr
tmpfs 32G 73M 32G 1% /dev/shm
tmpfs 5.0M 4.0K 5.0M 1% /run/lock
efivarfs 128K 47K 77K 38% /sys/firmware/efi/efivars
tmpfs 32G 0 32G 0% /run/qemu
/dev/mapper/vgubuntu-lv_tmp 9.8G 19M 9.3G 1% /tmp
/dev/mapper/vgubuntu-lv_var 20G 14G 4.6G 76% /var
/dev/mapper/vgdata-lvdata 1.9T 1.4T 353G 81% /data
/dev/mapper/vgubuntu-lv_nfs_stub 3.8M 72K 3.4M 3% /nfs
/dev/mapper/vgubuntu-lv_home 492G 146G 321G 32% /home
/dev/mapper/vgubuntu-lv_var_log 2.0G 598M 1.3G 32% /var/log
/dev/mapper/vgubuntu-lv_var_tmp 974M 622M 285M 69% /var/tmp
/dev/mapper/vgubuntu-lv_var_audit 488M 24K 452M 1% /var/log/audit
/dev/nvme0n1p1 511M 6.1M 505M 2% /boot/efi
kaiju:/t3/cvw 15T 6.2T 7.6T 45% /nfs/cvw
worknas:/volume1/homes 3.6T 1.4T 2.3T 39% /nfs/worknas/homes
media:/volume1/music 13T 6.6T 6.1T 52% /nfs/media/music
media:/volume1/video 13T 6.6T 6.1T 52% /nfs/media/video
worknas:/volume1/Work 3.6T 1.4T 2.3T 39% /nfs/worknas/Work
kaiju:/t3/cvo 15T 6.2T 7.6T 45% /nfs/cvo
worknas:/volume1/repositories 3.6T 1.4T 2.3T 39% /nfs/worknas/repositories
tmpfs 6.3G 144K 6.3G 1% /run/user/1000
"apt update" runs normally, but when I run an "apt upgrade -y" I get the following:
root@jaeger:~# apt upgrade -y
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
snapd
The following packages will be upgraded:
gir1.2-vte-2.91 gnome-remote-desktop libvte-2.91-0 libvte-2.91-common linux-firmware
5 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
2 not fully installed or removed.
3 standard LTS security updates
Need to get 267 MB of archives.
After this operation, 1,549 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-firmware all 20220329.git681281e4-0ubuntu3.31 [267 MB]
Get:2 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libvte-2.91-0 amd64 0.68.0-1ubuntu0.1 [214 kB]
Get:3 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libvte-2.91-common amd64 0.68.0-1ubuntu0.1 [9,408 B]
Get:4 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 gir1.2-vte-2.91 amd64 0.68.0-1ubuntu0.1 [10.7 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu jammy-updates/main amd64 gnome-remote-desktop amd64 42.9-0ubuntu0.22.04.2 [128 kB]
Fetched 267 MB in 6s (46.3 MB/s)
(Reading database ... 237804 files and directories currently installed.)
Preparing to unpack .../linux-firmware_20220329.git681281e4-0ubuntu3.31_all.deb ...
Unpacking linux-firmware (20220329.git681281e4-0ubuntu3.31) over (20220329.git681281e4-0ubuntu3.30) ...
Preparing to unpack .../libvte-2.91-0_0.68.0-1ubuntu0.1_amd64.deb ...
Unpacking libvte-2.91-0:amd64 (0.68.0-1ubuntu0.1) over (0.68.0-1) ...
Preparing to unpack .../libvte-2.91-common_0.68.0-1ubuntu0.1_amd64.deb ...
Unpacking libvte-2.91-common (0.68.0-1ubuntu0.1) over (0.68.0-1) ...
Preparing to unpack .../gir1.2-vte-2.91_0.68.0-1ubuntu0.1_amd64.deb ...
Unpacking gir1.2-vte-2.91:amd64 (0.68.0-1ubuntu0.1) over (0.68.0-1) ...
Preparing to unpack .../gnome-remote-desktop_42.9-0ubuntu0.22.04.2_amd64.deb ...
Unpacking gnome-remote-desktop (42.9-0ubuntu0.22.04.2) over (42.9-0ubuntu0.22.04.1) ...
Setting up initramfs-tools (0.140ubuntu13.4) ...
update-initramfs: deferring update (trigger activated)
Setting up libvte-2.91-common (0.68.0-1ubuntu0.1) ...
Setting up linux-firmware (20220329.git681281e4-0ubuntu3.31) ...
update-initramfs: Generating /boot/initrd.img-6.5.0-26-generic
cpio: write error: No space left on device
E: mkinitramfs failure cpio 2
update-initramfs: failed for /boot/initrd.img-6.5.0-26-generic with 1.
dpkg: error processing package linux-firmware (--configure):
installed linux-firmware package post-installation script subprocess returned error exit status 1
Setting up libvte-2.91-0:amd64 (0.68.0-1ubuntu0.1) ...
Setting up gnome-remote-desktop (42.9-0ubuntu0.22.04.2) ...
Setting up gir1.2-vte-2.91:amd64 (0.68.0-1ubuntu0.1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.8) ...
Processing triggers for libglib2.0-0:amd64 (2.72.4-0ubuntu2.3) ...
Processing triggers for initramfs-tools (0.140ubuntu13.4) ...
update-initramfs: Generating /boot/initrd.img-6.5.0-26-generic
cpio: write error: No space left on device
E: mkinitramfs failure cpio 2
update-initramfs: failed for /boot/initrd.img-6.5.0-26-generic with 1.
dpkg: error processing package initramfs-tools (--configure):
installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
linux-firmware
initramfs-tools
E: Sub-process /usr/bin/dpkg returned an error code (1)
Listing the Linux Kernels, I see plenty of old config files, but the only fully installed kernel is 6.5.0-26. I got partial installs on 27, 28 and 35 at one point, but removed them in an effort to fix this problem. 6.5.0-26 is the only recent kernel that has worked.
dpkg --list | grep linux-image
ic linux-image-5.15.0-25-generic 5.15.0-25.25 amd64 Signed kernel image generic
ic linux-image-5.15.0-58-generic 5.15.0-58.64 amd64 Signed kernel image generic
ic linux-image-5.15.0-60-generic 5.15.0-60.66 amd64 Signed kernel image generic
ic linux-image-5.19.0-32-generic 5.19.0-32.33~22.04.1 amd64 Signed kernel image generic
ic linux-image-5.19.0-35-generic 5.19.0-35.36~22.04.1 amd64 Signed kernel image generic
ic linux-image-5.19.0-38-generic 5.19.0-38.39~22.04.1 amd64 Signed kernel image generic
ic linux-image-5.19.0-40-generic 5.19.0-40.41~22.04.1 amd64 Signed kernel image generic
ic linux-image-5.19.0-41-generic 5.19.0-41.42~22.04.1 amd64 Signed kernel image generic
ic linux-image-5.19.0-42-generic 5.19.0-42.43~22.04.1 amd64 Signed kernel image generic
ic linux-image-5.19.0-43-generic 5.19.0-43.44~22.04.1 amd64 Signed kernel image generic
ic linux-image-5.19.0-46-generic 5.19.0-46.47~22.04.1 amd64 Signed kernel image generic
ic linux-image-5.19.0-50-generic 5.19.0-50.50 amd64 Signed kernel image generic
rc linux-image-6.2.0-26-generic 6.2.0-26.26~22.04.1 amd64 Signed kernel image generic
rc linux-image-6.2.0-31-generic 6.2.0-31.31~22.04.1 amd64 Signed kernel image generic
rc linux-image-6.2.0-32-generic 6.2.0-32.32~22.04.1 amd64 Signed kernel image generic
rc linux-image-6.2.0-33-generic 6.2.0-33.33~22.04.1 amd64 Signed kernel image generic
rc linux-image-6.2.0-34-generic 6.2.0-34.34~22.04.1 amd64 Signed kernel image generic
rc linux-image-6.2.0-35-generic 6.2.0-35.35~22.04.1 amd64 Signed kernel image generic
rc linux-image-6.2.0-36-generic 6.2.0-36.37~22.04.1 amd64 Signed kernel image generic
rc linux-image-6.2.0-37-generic 6.2.0-37.38~22.04.1 amd64 Signed kernel image generic
rc linux-image-6.2.0-39-generic 6.2.0-39.40~22.04.1 amd64 Signed kernel image generic
rc linux-image-6.5.0-14-generic 6.5.0-14.14~22.04.1 amd64 Signed kernel image generic
rc linux-image-6.5.0-15-generic 6.5.0-15.15~22.04.1 amd64 Signed kernel image generic
rc linux-image-6.5.0-17-generic 6.5.0-17.17~22.04.1 amd64 Signed kernel image generic
rc linux-image-6.5.0-18-generic 6.5.0-18.18~22.04.1 amd64 Signed kernel image generic
rc linux-image-6.5.0-21-generic 6.5.0-21.21~22.04.1 amd64 Signed kernel image generic
ic linux-image-6.5.0-25-generic 6.5.0-25.25~22.04.1 amd64 Signed kernel image generic
ii linux-image-6.5.0-26-generic 6.5.0-26.26~22.04.1 amd64 Signed kernel image generic
ic linux-image-6.5.0-27-generic 6.5.0-27.28~22.04.1 amd64 Signed kernel image generic
ic linux-image-6.5.0-28-generic 6.5.0-28.29~22.04.1 amd64 Signed kernel image generic
ic linux-image-6.5.0-35-generic 6.5.0-35.35~22.04.1 amd64 Signed kernel image generic
I've tried multiple things to resolve this, and I've restored the system from backup many times. Things I've tried:
- "apt autoremove" and "apt --purge autoremove" will not get rid of the old Kernel config files, mkintramfs fails with the cpio 2 and initramfs-tools shows the post-installation return code of 1
- Forcing reconfig with dpkg
dpkg --configure -a
Setting up initramfs-tools (0.140ubuntu13.4) ...
update-initramfs: deferring update (trigger activated)
Setting up linux-firmware (20220329.git681281e4-0ubuntu3.31) ...
update-initramfs: Generating /boot/initrd.img-6.5.0-26-generic
cpio: write error: No space left on device
E: mkinitramfs failure cpio 2
update-initramfs: failed for /boot/initrd.img-6.5.0-26-generic with 1.
dpkg: error processing package linux-firmware (--configure):
installed linux-firmware package post-installation script subprocess returned error exit status 1
Processing triggers for initramfs-tools (0.140ubuntu13.4) ...
update-initramfs: Generating /boot/initrd.img-6.5.0-26-generic
cpio: write error: No space left on device
E: mkinitramfs failure cpio 2
update-initramfs: failed for /boot/initrd.img-6.5.0-26-generic with 1.
dpkg: error processing package initramfs-tools (--configure):
installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
linux-firmware
initramfs-tools
- Purged all old kernels manual with apt --purge linux-image-version.numbers-generic. Purged the newer 27-35 kernels that way, too. Kernels were removed successfully, but didn't fix issue. Restored from backup since then.
- Same thing, but with dpkg instead of apt
- Moved main partition, created 1GB ext4 partition between EFI and main partition and assigned to boot. Moved stuff out of /boot on / and into /boot2, edited /etc/fstab to mount /boot then /boot/efi. Booted, but did not solve issue, restored from backup.
- Reviewed /var/log/apt/history.log and term.log, didn't see any more detailed info
- Reviewed dmesg, didn't see anything.
- Ran "dmesg -w" while running "apt update" and "apt upgrade". While running "apt update" I saw several
audit: type=1400 audit(1718299437.000:129): apparmor="DENIED" operation="open" class="file" profile="ubuntu_pro_esm_cache//dpkg" name="/var/lib/dpkg/arch" pid=70226 comm="dpkg" requested_mask="r" denied_mask="r" fsuid=0 ouid=0, but nothing during the "apt upgrade". Ran "systemctl stop apparmor" to disable apparmor and tried the update again, no dmesg errors but still no joy. - Several variations of purging / reinstalling linux-firmware. It never goes well, invariably revert to backup.
==== NEW INFO 6/13/24 16:18 MT:
I've cleaned up all the old kernels with the following commands:
dpkg --purge $(dpkg -l | awk '/^rc/{print $2}')
dpkg --purge $(dpkg -l | awk '/^ic/{print $2}')
Now only one Kernel (dangerous, I know, but it was the only working one anyway). dpkg --configure -a shows me the same errors. This is the most concise way I know to show the problem:
root@jaeger:~# dpkg --list | grep linux-image
ii linux-image-6.5.0-26-generic 6.5.0-26.26~22.04.1 amd64 Signed kernel image generic
root@jaeger:~# dpkg --configure -a
Setting up initramfs-tools (0.140ubuntu13.4) ...
update-initramfs: deferring update (trigger activated)
Setting up linux-firmware (20220329.git681281e4-0ubuntu3.31) ...
update-initramfs: Generating /boot/initrd.img-6.5.0-26-generic
cpio: write error: No space left on device
E: mkinitramfs failure cpio 2
update-initramfs: failed for /boot/initrd.img-6.5.0-26-generic with 1.
dpkg: error processing package linux-firmware (--configure):
installed linux-firmware package post-installation script subprocess returned error exit status 1
Processing triggers for initramfs-tools (0.140ubuntu13.4) ...
update-initramfs: Generating /boot/initrd.img-6.5.0-26-generic
cpio: write error: No space left on device
E: mkinitramfs failure cpio 2
update-initramfs: failed for /boot/initrd.img-6.5.0-26-generic with 1.
dpkg: error processing package initramfs-tools (--configure):
installed initramfs-tools package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
linux-firmware
initramfs-tools
FYI, also updated BIOS on my MSI B550A-PRO from BIOS E7C56AMS.AE1 (06/09/2023) to E7C56AMS.AI1 (04/30/2024) earlier today, no change. The old BIOS was in place long before my problems started.
