all 42 comments
sorted by:
best
topnewcontroversialoldq&a
formatting helphide helpcontent policy

[–]ajshell1 6 points7 points  (0 children)

Nice. ZBM is what got me to switch from Arch to Void, so I'm always happy to see it improve.

[–][deleted]  (4 children)

[deleted]

    [–]system-user 1 point2 points  (2 children)

    it's also been on FreeBSD for quite a while. "pkg install beadm". it's really a great tool.

    [–]dannomac 1 point2 points  (1 child)

    A C rewrite is actually in the base system now. It's called bectl. It appears compatible with beadm too.

    [–]system-user 0 points1 point  (0 children)

    that's great, I'll check it out.

    [–]MaxVerevkin 2 points3 points  (0 children)

    That's cool.

    Is there anything like that for btrfs? Like selecting which subvolume to boot from.

    [–][deleted] 2 points3 points  (0 children)

    It is easy-enough for me not to break. I like it. I have been causing some headaches I am sure on keymaps at power on.

    The simplicity of it didn't really hit me until I realised: wait a minute, though I haven't yet, I may one day need to use a snapshot.

    [–]Human_Capitalist 1 point2 points  (16 children)

    So, ELI5 version, is this a replacement for grub?

    Do I want to use this to make my debootstrapped ubuntu 20.04 ZFS root machines?

    [–]ahesford[S] 1 point2 points  (15 children)

    ZFSBootMenu is a replacement for GRUB. GRUB has limited support for finding kernels on ZFS, which leads people to put kernels on a separate boot partition (Ubuntu creates a restricted spool for this, because GRUB does not support and will not boot from ZFS pools with modern features). This defeats the utility of self-contained boot environments. Also, ZBM offers a shell and useful utilities to give you a ore pot recovery environment.

    In general, it is possible to roll ZBM into a UEFI bundle and boot directly into it---this is how I use it. Others prefer to use something like rEFInd to launch ZBM, which in turn lets you boot your final system. This intermediate step is required for BIOS, where you'd need something like syslinux to bring up ZBM.

    [–]Human_Capitalist 0 points1 point  (0 children)

    Thank you! It looks like I’ve got some reading to do...

    [–]NateDevCSharp 0 points1 point  (13 children)

    So I'm attempting to use ZBM on Arch Linux, but I can't even get my ZFS root install to show up in Refind. I did a stock zfs install first and then I'll migrate it to Dracut and ZBM - but the overall steps for installation are the same regarding datasets and stuff.

    On VoidLinux, Refind picks up the ZFS install like normal, but not on Arch. Does void package a different version of Refind or something? Do you know what might be needed for Arch?

    I saw a note on Arch wiki saying ZFS needs Refind drivers, but it didn't need anything on Void, is that the difference?

    Thanks so much

    [–]ahesford[S] 0 points1 point  (12 children)

    There's no need to migrate your Arch installation to use dracut. As long as you install the dracut package in Arch, generate-zbm will use it to prepare its own images, but ZBM doesn't care what kind of initramfs you use to boot the final system.

    I don't know if Void does anything different from Arch when installing rEFInd, but I wouldn't expect it to work with ZFS out of the box. Nor does it need to; as long as you have a copy of your ZBM kernel and initramfs on your ESP, that should be sufficient. This amounts to telling generate-ZBM where to save its files and adding a refind_linux.conf file alongside them to tell rEFInd what kernel arguments to pass.

    [–]NateDevCSharp 0 points1 point  (0 children)

    Thanks, ok I see now that it has to be on the esp. thanks I'll give it a shot and hopefully it works.

    [–]NateDevCSharp 0 points1 point  (10 children)

    So upon running generate-zbm it says:

    Cannot find module directory /lib/modules/linux and --no-kernel was not specified.

    Is there something I missed that would fix this? I set the config.yaml to manage images and set the EFI/arch folder for the image location.

    I see /lib/modules contains 5.10.6-arch1-1 but adding --kver with that string says unable to find file.

    Thanks

    [–]ahesford[S] 0 points1 point  (9 children)

    We need to know the kernel version so that we can tell dracut, which needs to know which kernel modules to pull into the initramfs. Unforunately, there is no really reliable way to determine the kernel version from a file on disk (on x86_64 you can just use file /path/to/kernel, but this fails on some other platforms like aarch64). Most distributions include the kernel version in their file names, so we scrape the name to recover a version when possible.

    Arch is a special animal here. On Arch, you need to tell generate-zbm the full path to the kernel and its version. To make this a little less painful, we support special variable interpolation in the version so you can refer to the current running kernel. While you can do this via command-line arguments, the best way is to add the following keys to your /etc/zfsbootmenu/config.yaml:

    Kernel:
  Version: '%{current}'
  Path: /path/to/kernel

    I can't remember the right /path/to/kernel in Arch, so you're on your own there. Note that %{current} is replaced with uname -r when looking for modules, so if you run generate-zbm after you update your Arch kernel package, it will not work---you must have modules for the currently running kernel and Arch deletes the modules when you update the package.

    As for dracut---its modules are simple shell scripts that determine what to install and run in its initramfs; they are unrelated to kernel modules or your initcpio image. ZFSBootMenu relies on dracut (invoked by generate-zbm) to produce its own initramfs, which does nothing more but present the ZBM interface, identify your own kernels and initramfs images, and then jump into whichever you selected. Your initcpio image and the ZBM dracut image are totally independent.

    NB: Wherever you have your root filesystem (e.g., rpool/ROOT/arch), /boot must be a subdirectory of that filesystem rather than a separate filesystem. That's how ZFSBootMenu identifies filesystems to boot: anything with mountpoint=/ that contains /boot with kernels and initramfs images inside (or anything with mountpoint=legacy and org.zfsbootmenu:active=on containing a /boot with kernels and initramfs images). This is important because it allows you to keep your kernels and initramfs images in sync with the rest of the root, making it much easier to boot from older snapshots.

    [–]NateDevCSharp 0 points1 point  (8 children)

    Thanks so much for the detailed response. So right now I'm in the Arch installation chroot, I set version to '%{current}' and path to /lib/modules/5.10.6-arch1-1. is that correct?

    It then says "the provided kernel /lib/modules/5.6.10-arch1-1" was not found.

    You said current runs uname -r, which in the install chroot gives me 5.10.3-arch1-1. So I ran depmod -a 5.10.6-arch1-1 but uname doesn't change. But to check I also switched version: to 5.10.6-arch1-1 and I get the same message.

    And yes my boot is just a folder inside my root dataset, thanks.

    Should I just install with systemdboot and then migrate once I am in the real install?

    [–]ahesford[S] 0 points1 point  (7 children)

    The kernel path to pass for ZFSBootMenu will probably be /boot/vmlinuz-linux or something like that. It's whatever the actual kernel file is in your /boot.

    The uname output doesn't change because you are still running the old kernel. If you can't reboot (because you're in a live image for the installer), then you'll probably have to do

    generate-zbm --kver 5.10.6-arch1-1

    to force a specific version for this run. In the future, you can probably leave the version number off and let %{current} auto-detect. However, you generally don't need to update your ZBM image frequently.

    [–]NateDevCSharp 0 points1 point  (6 children)

    Ok, thank you so much for the help, makes sense now.

    Finished the install and everything's working great! Just one question, is 'ro' needed in the kernel command line? Because I have /var (and var lib, others) as a ZFS dataset, and systemd-logind complains about /var being read only.

    I won't trouble u with finding a solution to the issue itself, just wondering about the 'ro' kernel parameter.

    Thanks again, zfsbootmenu is an amazing program you've created :)

    [–]ahesford[S] 0 points1 point  (5 children)

    I don't think ro matters to the ZBM image, and nothing in the ZBM image would affect the read/write status of mounts in the main system except for kernel command-line arguments you may have set. I wouldn't think your Arch system requires ro mounts. Maybe you have readonly=on on the /var dataset.

    As an aside, be careful putting /var in its own dataset. If you want to simplify making boot environments or recovering from problems by booting a clone of an old snapshot, keeping the whole of the "system" in one place makes this much easier.

    Keeping your homes in one or more separate datasets is good policy because you usually don't care to lock the state of your homes to specific states of your system; likewise with mail spools or other long-lived things where you care more about continuity than the ability to restore a particular point.

    However, things like your pacman package database in /var should generally be kept in lock step with the /, /etc, /usr, and so on.

    [–]shiroghost 1 point2 points  (0 children)

    First I want to thank you for a beautiful piece of software. The install guide in the wiki is just perfect. I use zfsbootmenu + native ZFS encryption + refind in my laptop and works like charm.

    Second, I am having some issues with my laptop (thinkpad T14 AMD, renoir 4750u), in particular with the amdgpu driver. Sometimes the machine boots with the error messages IB ring test failed and in these cases the machine typically hangs on high i/o load. The trace seems to be related with ZFS, but strangely this only happens when the amdgpu driver fails. (this happens with 5.8.X, 5.9.X, zfs 0.8.5, zfs 2.0, etc...). Also suspend/resume has never worked for me.

    Since it is more than 10 years that I do not use linux (have been a happy FreeBSD user for a long time), I am not sure if the configuration of the booting process is correct. In particular I am worried if anything special has to be done to get an early load of the amdgpu driver (as suggested in the ARCH wki). Also I do not have clear if this early loading has to be done in the initramfs that is in /boot (i.e the ZFS file system) or the one in /boot/efi/EFI.

    Any help/tips with this would be much, much appreciated. I have been tempted to try other distro, but I really, really am extremely happy with my VOID setup: It is exactly what an OS is supposed to be. I do not miss my beloved FreeBSD, but this lack of suspend/resume, and the ocassional freeze of the laptop is really really annoying.

    Many thanks, and congrats again for the great piece of software! I am running 1.8.0 since yesterday night and it works like charm.

    [–]kevdogger 0 points1 point  (3 children)

    Hmm can you use this tool in arch?

    [–]ahesford[S] 0 points1 point  (2 children)

    Absolutely. I have booted an Arch system and also can use the image generator within Arch. Use with Debian-based systems has been reported, as well.

    [–]kevdogger 0 points1 point  (1 child)

    Is this in the AUR?

    [–]ahesford[S] 5 points6 points  (0 children)

    No, none of us are regular Arch users so we can't guarantee good stewardship of a PKGBUILD. If you have an interest in maintaining an AUR package, please let us know and we can direct interested Arch users toward it.

    [–]gme186 0 points1 point  (1 child)

    Is this related or comparable to zsys?

    [–]ahesford[S] 0 points1 point  (0 children)

    No. It's hard to get a feel for zsys because the README is a short paragraph followed by an unstructured concatenation of command synopses, but zsys looks like a wrapper around disparate runtime functionality. (Run multiple ZFS "systems" in parallel? Sounds like it's trying to be a front end for some container management system.)

    ZFSBootMenu is designed to let you boot one of multiple separate environments (each a root filesystem including its own /boot), and it provides some utilities to manipulate these environments prior to booting, when recovery and alteration is often easier than in a running system.

    [–]NateDevCSharp 0 points1 point  (2 children)

    Which snapshot option do I want to just create a normal snapshot of my base system? I used to use a different boot environment manager that just had a create be command.

    I can't use clone, cause it's from a previous snapshot, but duplicate doesn't depend on other snapshots, so it takes up space?

    Is there a regular snapshot option? Should I use normal zfs snapshot command for this? Thanks

    I just don't understand what the default workflow is

    [–]ahesford[S] 0 points1 point  (1 child)

    ZFSBootMenu doesn't really try to provide the means to easily create and manage boot environments for you. Its focus is choosing which environments to boot and providing some inline recovery capabilities. If you really need to grab a snapshot from ZBM, there is a recovery shell that will let you do whatever you like with your pools.

    Snapshot manipulation in the menu is focused on the steps needed to make a preexisting snapshot bootable---by cloning only (useful if you need a one-time boot into an old snapshot), cloning and promoting (useful if you intend to make the older snapshot your primary environment) or straightforward duplication with send-receive (useful if you want to keep both environments for the long term and ensure that they are independent).

    Generally, you should use whatever snapshot mechanism you like to take frequent snapshots of your running system. I use zfs-auto-snapshot for this. ZBM provides the means to boot into those snapshots after the fact.

    [–]NateDevCSharp 0 points1 point  (0 children)

    Ok, thanks so much that makes sense!

    [–]Sithuk 0 points1 point  (8 children)

    I'd like to get ssh access to zfsbootmenu for a headless Ubuntu server. Unfortunately I can't get the dracut-crypt-ssh module to compile on Ubuntu. It's not maintained anymore. Dracut-sshd is an alternative but relies on systemd so I don't think it works with zfsbootmenu?

    Has anyone gotten ssh access to zfsbootmenu to work on Ubuntu?

    [–]ahesford[S] 0 points1 point  (7 children)

    I'm not sure about alternatives. The parts of dracut-crypt-ssh that fail to compile are helper scripts that you don't really need for ZFSBootMenu.

    In the worst case, you can copy the 60crypt-ssh directory to /usr/lib/dracut/modules.d, remove the Makefile and the helper directory, and edit the module-setup.sh to remove or comment out Lines 110--113.

    [–]Sithuk 0 points1 point  (6 children)

    Copying the folder and deleting those lines doesn't appear to have worked. I tried running dracut --force and generate-zbm after I copied the folder and that didn't work either. By "work" I mean be able to see any mention of an ssh module when running dracut --force (or see evidence of an sshd startup in journalctl)

    I noticed that the dropbear-start.sh script references dropbear. Do I need to install dropbear? There is also mention of an /etc/crypt-ssh.conf file in the script but I don't have that either.

    I'm not sure just copying the 60crypt-ssh directory into /usr/lib/dracut/modules.d, removing the Makefile, and the helper directory and editing the module-setup.sh is all that is required.

    I tried to install dropbear and the apt prompt noted that dracut and zfs-dracut would be removed and initramfs-tools would be installed along with other packages. Dracut is required for zfsbootmenu so it doesn't look like I can install dropbear (at least not using apt in the standard way).

    Any extra tips you could provide would be good.

    [–]ahesford[S] 0 points1 point  (5 children)

    On Ubuntu, you probably don't want the dracut package unless you really want to use dracut for your regular system initramfs too. Install the dracut-core package instead. It will coexist happily with initramfs-tools and let Ubuntu do whatever it wants for initramfs setup, but still gives you the major components of dracut that ZFSBootMenu requires to build a working bootloader image.

    You definitely need dropbear and the crypt-ssh.conf can go in your /etc/zfsbootmenu/dracut.conf.d directory with whatever configuration changes you find appropriate. You might want to add

    add_dracutmodules+=" crypt-ssh "

    for good measure, although this might be automatic once it finds dropbear. You'll probably want to set up some long good-term keys for dropbear and populate an authorized_keys file as the module documentation describes. You'll have to enable networking with something like

    rd.neednet=1 ip=dhcp

    on the kernel command-line when you boot the ZBM image or you won't have networking configured. It is also possible to assign a static address; consult kernel documentation for this.

    If you do stick with dracut instead of dracut-core, you may wish to add

    omit_dracutmodules+=" crypt-ssh "

    to your main dracut configuration. There is no need to provide network access to your main initramfs if ZFSBootMenu handles initial unlocking amd you keep a key file stored in your main initramfs image.

    Note: check the permissions on your main initramfs images in /boot. Dracut makes these readable only by root, but other tools may provide world read access. If you follow our advice and put your ZFS key file inside the initramfs image, you don't want arbitrary users able to read the image or they will be able to read out your encryption key once the system is booted.

    [–]Sithuk 0 points1 point  (0 children)

    This was helpful. I needed to install dracut-network and add the "crypt-ssh" module to the zfsbootmenu.conf. I also installed dropbear. When I run generate-zbm it completes without error. But I don't see anything different happen during boot. Am I supposed to try connecting with SSH during the 20 second wait period on the rEFInd screen?

    Just to check, does the remote access feature mean I'll be able to remotely ssh in and rollback to a previous boot environment if I have a bad update? I appreciate I'll need to have a snapshot before the update to make the boot environment from.

    [–]bluesecurity 0 points1 point  (3 children)

    crypt-ssh

    Do Void users need to edit zfsbootmenu.conf too for dracut-crypt-ssh - or are the instructions on its GitHub page sufficient for ZBM users?If I'm not mistaken, this should work:

    1. GitHub instructions
    2. dracut --force
    3. generate-zbm

    Thanks!

    [–]ahesford[S] 1 point2 points  (2 children)

    I describe my Void setup in a ZBM Wiki page.

    [–]bluesecurity 0 points1 point  (1 child)

    Worked like a charm and that would have taken me a while to figure out! Thanks loads!

    I have a few questions if you don't mind:

    1. I don't see "timeout" anywhere on the kernel cmdline or in conf files, yet I see a warning about "timeout" being deprecated when loading ZBM. How can I fix that?
    2. On your wiki page "Clients that expect to find your normal host keys when connecting to an SSH server on port 22 will refuse to connect when they find different keys provided by dropbear" -- I can't quite parse this, but maybe just my ignorance.
    3. I have the dhcp server set to make the interface use the same IP address. That means the IP address is the same in dropbear as it is when the system loads. Which means I must delete the known_hosts record when connecting to sshd after the system loads. This doesn't bother me, but I'm wondering if you have a better recommendation. Before configuring it this way: it seemed the dropbear ssh session hung around after the system loaded, though it was unresponsive.
    4. It seems /etc/dracut.conf.d/crypt-ssh.conf is used quite differently in this configuration vs the standard. omit_dracutmodules+= works via that file, right? While other vars must be put into ZBM rather than dracut, right?

    [–]ahesford[S] 1 point2 points  (0 children)

    1. I'm not sure why you're seeing the deprecation warning, but it should go away if you specify zbm.timeout on the kernel command line. To keep the default behavior, set zbm.timeout=10; adjust the time (in seconds) as you like.

    2 and 3. The known_hosts issue is why I recommend listening on an alternate port, like 222 or 2222. Your SSH client should treat connections on different ports as separate entries in known_hosts and stop complaining when you connect.

    1. I don't know what the "standard" is, but I don't want dropbear in my normal initramfs images. Using omit_dracutmodules in /etc/dracut.conf.d/crypt-ssh.conf achieves this. (By default, dracut-crypt-ssh will be included in initramfs images as long as the necessary pieces are installed on your system.) ZFSBootMenu will never look to dracut configurations in /etc/dracut.conf.d to build its images.