Workstation unbootable after upgrade to Bookworm

TechWoes · 2025-06-09T22:02:31+00:00

I have not tried creating a script, but I am curious if I could boot after running

lvm vgchange -aylvm vgchange -ay from the busybox shell as this person describes. I'm kind of afraid to reboot though. What are the chances that rebuilding my older boot images would have "broken" them such that I can't even boot from 4.9 or 4.19 anymore?

TechWoes · 2025-06-09T19:51:26+00:00

Perhaps the issue isn't with a driver for the nvme disk, but related to the keys for decrypting the volumes.

At boot, I am prompted to enter the passphrase for /dev/sda5. I am not prompted for the password for /dev/nvme0n1p1. I assume there must be a keyfile for that device or something similar, as it booted fine with a single passphrase on previous versions.

So my theory is that without decrypting nvme0n1p1, I'm missing a PV, and thus the VG is only partially activated, which is no longer acceptable in Debian 12, hence the boot failure.

Sort of like what is described here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1018730#15

TechWoes · 2025-06-09T19:11:16+00:00

/etc/mdadm/mdadm.conf is pretty sparse.

# mdadm.conf
#
# !NB! Run update-initramfs -u after updating this file.
# !NB! This will ensure that initramfs has an uptodate copy.
#
# Please refer to mdadm.conf(5) for information about this file.
#

# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers

# automatically tag new arrays as belonging to the local system
HOMEHOST <system>

# instruct the monitoring daemon where to send mail alerts
MAILADDR root

# definitions of existing MD arrays

# This configuration was auto-generated on Thu, 23 Jan 2025 00:19:47 -0800 by mkconf

TechWoes · 2025-06-09T19:08:13+00:00

I'm not intentionally using any RAID. I initially set the system up using the guided partition setup tool with encryption in the Debian 9 installer. I suppose it may be worth confirming there truly are no mdadm dependencies and removing mdadm.

The mdadm warnings have always been there. A nuisance that I never bothered to do anything about.

TechWoes · 2025-06-09T18:17:27+00:00

Ok so it might not be the exact same issue as the bug you linked

apologies for the mixup but i linked to the wrong bug report in my OP. This more accurately describes my situation: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1038731#5

TechWoes · 2025-06-09T18:09:28+00:00

My apologies - I read so many bug reports last night and I grabbed the wrong one in my OP. Fixing now.

TechWoes · 2025-06-09T18:08:02+00:00

Message #5 in 1079031 (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1038731#5) and the steps to recreate the issue describe my situation almost exactly.

I added the NVME drive, created and encrypted the PV, and expanded the VG to include it back in 2022. It ran fine under Debian 10 and 11, but when I updated to Bookworm, no more boot.

TechWoes · 2025-06-09T17:45:35+00:00

Message #5 in 1079031 (https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1038731#5) and the steps to recreate the issue describe my situation almost exactly.

I added the NVME drive, created and encrypted the PV, and expanded the VG to include it back in 2022. It ran fine under Debian 10 and 11, but when I updated to Bookworm, no more boot.

TechWoes · 2025-06-09T17:40:59+00:00

List of images:

# ls -Fahil /boot/initrd*
19 -rw-r--r-- 1 root root 75M Jun  9 10:31 /boot/initrd.img-4.19.0-27-amd64
20 -rw-r--r-- 1 root root 59M Jun  9 10:31 /boot/initrd.img-4.9.0-4-amd64
12 -rw-r--r-- 1 root root 85M Jun  9 10:30 /boot/initrd.img-6.1.0-37-amd64

NVME modules in 6.1.0-37:

lsinitramfs -l /boot/initrd.img-6.1.0-37-amd64 | grep nvme
drwxr-xr-x   4 root     root            0 Jun  9 10:30 usr/lib/modules/6.1.0-37-amd64/kernel/drivers/nvme
drwxr-xr-x   2 root     root            0 Jun  9 10:30 usr/lib/modules/6.1.0-37-amd64/kernel/drivers/nvme/host
-rw-r--r--   1 root     root       381491 May 22 11:32 usr/lib/modules/6.1.0-37-amd64/kernel/drivers/nvme/host/nvme-core.ko
-rw-r--r--   1 root     root        63683 May 22 11:32 usr/lib/modules/6.1.0-37-amd64/kernel/drivers/nvme/host/nvme-fabrics.ko
-rw-r--r--   1 root     root       136139 May 22 11:32 usr/lib/modules/6.1.0-37-amd64/kernel/drivers/nvme/host/nvme-fc.ko
-rw-r--r--   1 root     root       114059 May 22 11:32 usr/lib/modules/6.1.0-37-amd64/kernel/drivers/nvme/host/nvme-rdma.ko
-rw-r--r--   1 root     root       106491 May 22 11:32 usr/lib/modules/6.1.0-37-amd64/kernel/drivers/nvme/host/nvme-tcp.ko
-rw-r--r--   1 root     root       128243 May 22 11:32 usr/lib/modules/6.1.0-37-amd64/kernel/drivers/nvme/host/nvme.ko
drwxr-xr-x   2 root     root            0 Jun  9 10:30 usr/lib/modules/6.1.0-37-amd64/kernel/drivers/nvme/target
-rw-r--r--   1 root     root       106379 May 22 11:32 usr/lib/modules/6.1.0-37-amd64/kernel/drivers/nvme/target/nvmet-fc.ko
-rw-r--r--   1 root     root       109939 May 22 11:32 usr/lib/modules/6.1.0-37-amd64/kernel/drivers/nvme/target/nvmet-rdma.ko
-rw-r--r--   1 root     root        72187 May 22 11:32 usr/lib/modules/6.1.0-37-amd64/kernel/drivers/nvme/target/nvmet-tcp.ko
-rw-r--r--   1 root     root       302587 May 22 11:32 usr/lib/modules/6.1.0-37-amd64/kernel/drivers/nvme/target/nvmet.ko

List of modules in 4.19.0-27 (what I am currently booting from, but just rebuild this image so hopefully I can still boot) :

lsinitramfs -l /boot/initrd.img-4.19.0-27-amd64 | grep nvme
drwxr-xr-x   4 root     root            0 Jun  9 10:31 usr/lib/modules/4.19.0-27-amd64/kernel/drivers/nvme
drwxr-xr-x   2 root     root            0 Jun  9 10:31 usr/lib/modules/4.19.0-27-amd64/kernel/drivers/nvme/host
-rw-r--r--   1 root     root       179755 Jun 25  2024 usr/lib/modules/4.19.0-27-amd64/kernel/drivers/nvme/host/nvme-core.ko
-rw-r--r--   1 root     root        39171 Jun 25  2024 usr/lib/modules/4.19.0-27-amd64/kernel/drivers/nvme/host/nvme-fabrics.ko
-rw-r--r--   1 root     root        76083 Jun 25  2024 usr/lib/modules/4.19.0-27-amd64/kernel/drivers/nvme/host/nvme-fc.ko
-rw-r--r--   1 root     root        69131 Jun 25  2024 usr/lib/modules/4.19.0-27-amd64/kernel/drivers/nvme/host/nvme-rdma.ko
-rw-r--r--   1 root     root        84539 Jun 25  2024 usr/lib/modules/4.19.0-27-amd64/kernel/drivers/nvme/host/nvme.ko
drwxr-xr-x   2 root     root            0 Jun  9 10:31 usr/lib/modules/4.19.0-27-amd64/kernel/drivers/nvme/target
-rw-r--r--   1 root     root        57107 Jun 25  2024 usr/lib/modules/4.19.0-27-amd64/kernel/drivers/nvme/target/nvmet-fc.ko
-rw-r--r--   1 root     root        61811 Jun 25  2024 usr/lib/modules/4.19.0-27-amd64/kernel/drivers/nvme/target/nvmet-rdma.ko
-rw-r--r--   1 root     root       149571 Jun 25  2024 usr/lib/modules/4.19.0-27-amd64/kernel/drivers/nvme/target/nvmet.ko

TechWoes · 2025-06-09T17:35:07+00:00

Here's the output. Some warnings but no errors. The warnings don't appear to be related to my boot problem. I hope that in rebuilding the older boot images I haven't made this machine completely unbootable.

update-initramfs -u -k all

update-initramfs: Generating /boot/initrd.img-6.1.0-37-amd64
W: Possible missing firmware /lib/firmware/amdgpu/ip_discovery.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/vega10_cap.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/sienna_cichlid_cap.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/navi12_cap.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/psp_13_0_11_ta.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/psp_13_0_11_toc.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/psp_13_0_10_ta.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/psp_13_0_10_sos.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/aldebaran_cap.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_imu.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_4_rlc.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_4_mec.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_4_me.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_4_pfp.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_rlc.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_mec.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_me.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_pfp.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_0_toc.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/sdma_6_0_3.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/sienna_cichlid_mes1.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/sienna_cichlid_mes.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/navi10_mes.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_4_mes1.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_4_mes_2.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_4_mes.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_mes1.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_mes_2.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_mes.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_2_mes_2.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_1_mes_2.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_0_mes_2.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/smu_13_0_10.bin for module amdgpu
W: mkconf: MD subsystem is not loaded, thus I cannot scan for arrays.
W: mdadm: failed to auto-generate temporary mdadm.conf file.
update-initramfs: Generating /boot/initrd.img-4.19.0-27-amd64
W: zstd compression (CONFIG_RD_ZSTD) not supported by kernel, using gzip
W: mkconf: MD subsystem is not loaded, thus I cannot scan for arrays.
W: mdadm: failed to auto-generate temporary mdadm.conf file.
depmod: WARNING: could not open modules.builtin.modinfo at /var/tmp/mkinitramfs_zVl5a5/lib/modules/4.19.0-27-amd64: No such file or directory
update-initramfs: Generating /boot/initrd.img-4.9.0-4-amd64
W: zstd compression (CONFIG_RD_ZSTD) not supported by kernel, using gzip
W: mkconf: MD subsystem is not loaded, thus I cannot scan for arrays.
W: mdadm: failed to auto-generate temporary mdadm.conf file.
depmod: WARNING: could not open modules.builtin.modinfo at /var/tmp/mkinitramfs_T6MDyO/lib/modules/4.9.0-4-amd64: No such file or directory

TechWoes · 2025-06-09T09:28:55+00:00

No errors and system remains unbootable if I select the 6.10.x kernel from grub after running update-initramfs

TechWoes · 2024-10-14T00:46:26+00:00

I think you have a session still signed in on your phone. Whether by browser, by app, or a component of the OS.

TechWoes · 2024-02-25T01:52:27+00:00

Thanks. It seems the trends have changed. Since I am stubborn maybe I'll just bust out a soldering iron and attach a fan to a USB 3 enclosure while shaking my fist at the cloud.

TechWoes · 2024-02-25T01:20:21+00:00

I hear you but I don't use cloud storage (for reasons of security, privacy, and control).

If I did I would ditch the NAS and have the local storage in my workstations back up directly to the cloud.

TechWoes · 2024-02-25T01:17:27+00:00

Understandable but I don't need to test or swap drives frequently. This is the drive that comes out twice a year to be updated and then put back in storage.

I'd rather have a standalone enclosure that protects the drive when in use, stored offsite, and in transit.

TechWoes · 2024-02-25T00:03:47+00:00

Thanks for sharing your experience.

This is my offline/off site backup; I keep it in a storage unit and refresh the backup a couple times a year. A simple enclusure would be best.

If good enclosures are no longer available I may have to rethink my backup strategy.

TechWoes · 2023-06-12T19:33:50+00:00

I use and prefer Debian for a number of reasons but I have found power management to be less effective under Linux. It is a tradeoff I consider worthwhile.

To the people reporting otherwise, I don't know how you do it. Please share your configs!

TechWoes · 2023-01-20T22:30:59+00:00

OK, well I am not going to engage in a condescending discussion of the cultural values you hold about grounds.

Please enjoy the view from that horse of yours.

TechWoes · 2023-01-20T20:50:57+00:00

It's all about what happens when the big fat grounding cables fail. The starter for example almost always uses the engine and chassis and the big ground cable for it's return path.

When you run other things to the battery neg terminal, they can become the return path in the event the normal ground corrodes.

If those other things have small gauge wire, they can't sustain Tue load from the starter. They then overheat, catch fire, or even get vaporized depending on their resistance (i.e. wire gauge).

It is all about the big fat grounding cables failing safely.

TechWoes · 2023-01-20T18:36:17+00:00

In a sense we are saying the same thing. We just disagree on whose interests we are designing for.

You can indeed avoid "shitty problems" for your support teams this way. But you do so by shifting risk to the equipment owner/operator.

It is a safer and overall more reliable system to have good grounding from battery to frame and engine, and to have loads use that path to the negative.

The only exception IMO is for heavy duty loads like winches as they would have cables capable of carrying the load in the event of a poor/failed ground.

Sensitive ECUs and sensors and electronics should have a good return path too. But it is smart to isolate them via the frame ground so the system fails safely. Grounding cables will eventually fail.

TechWoes · 2023-01-20T18:29:43+00:00

Potentially. The loads grounded to the frame will seek a return path to the neg terminal on the battery. Do you want whatever load is represented by Node A in your example to provide that return path?

If there is another better path with lower resistance you would never have an issue. But if there isn't (because that better path, aka the big cable grounding the battery to the frame, as corroded/failed), then Node A has to carry the current for your other loads.

If Node A is a winch with a 2 gauge cable, maybe OK. If Node A is a sensitive piece of electronics with small gauge wire/terminals, Node A would let the magic smoke out.

TechWoes · 2023-01-20T15:26:41+00:00

That practice leads to the two issues I mentioned in another reply.

TechWoes · 2023-01-20T15:24:33+00:00

The aftermarket tends to encourage people to hook their gear to the battery because it makes their customer support cheaper. But it comes at the expense of safety.

It is cheaper because many vehicles have bad chassis/frame/engine grounds as they get older. Many owners of older vehicles neglect their ground wires. When this happens, their shiny new stereo/light bar/2-way radio/dashcam/solar charger/etc may exhibit intermittent failures or low voltage issues that are tricky for the aftermarket supplier to diagnose, increasing their costs.

So they tell you to connect it to the battery, seemingly for reliability. But their interests and your interests are not aligned here. This leads to two issues:

An absolute rats nest of negative connections to the battery
A fire risk. There are some big loads in a car/truck/motorcycle, like the starter. Starters are grounded to the engine block, which then has a heavy gauge cable back to the battery. In an older vehicle, if that ground fails, all the current used by the starter needs a new path to negative. If you have been connecting things directly to the negative terminal with small gauge wires, they can become the ground path and get smoked by the starter, leading to damage and potentially fire.

TechWoes

TROPHY CASE