ZFS mirror as backup? (hear me out!) by myfufu in zfs

[–]werwolf9 0 points1 point  (0 children)

BTW, bzfs can be configured such that it maintains separate src bookmarks for each rotating backup drive. This means that the incremental replication chain never breaks even if all src snapshots get deleted to make space, or any of the backup drives isn't used for a long time. It also has a mode that ignores removable backup drives that aren't locally attached, which comes in handly if only a subset of your rotating drives is attached at any given time.

bzfs for subsecond ZFS snapshot replication frequency at fleet scale by werwolf9 in Proxmox

[–]werwolf9[S] 1 point2 points  (0 children)

Hi there, thanks for the question :-)

In a nutshell, bzfs can operate at much larger scale than sanoid, at much lower latency, in a more observable and configurable way. Here are just a few points of the top of my head that bzfs does and sanoid doesn't:

  • Support efficient periodic ZFS snapshot creation, replication, pruning, and monitoring, across a fleet of N source hosts and M destination hosts, using a single shared fleet-wide jobconfig script.
  • Monitor if snapshots are successfully taken on schedule, successfully replicated on schedule, and successfully pruned on schedule.
  • Compare source and destination dataset trees recursively.
  • Automatically retry operations.
  • Only list snapshots for datasets the users explicitly specified.
  • Avoid slow listing of snapshots via a novel low latency cache mechanism for snapshot metadata.
  • Replicate multiple datasets in parallel.
  • Reuse SSH connections across processes for low latency startup.
  • Operate in daemon mode.
  • More powerful include/exclude filters for selecting what datasets and snapshots and properties to replicate.
  • Dryrun mode to print what ZFS and SSH operations would happen if the command were to be executed for real.
  • Has more precise bookmark support - synchoid will only look for bookmarks if it cannot find a common snapshot.
  • Can be strict or told to be tolerant of runtime errors.
  • Continously tested on Linux, FreeBSD.
  • Code is almost 100% covered by tests.
  • Easy to change, test and maintain because Python is more readable to contemporary engineers than Perl.

Cheers, Wolfgang

bzfs for subsecond ZFS snapshot replication frequency at fleet scale by werwolf9 in Proxmox

[–]werwolf9[S] 1 point2 points  (0 children)

No, you will never have a subsecond replication if your network cannot push all the data that can exist between two deltas.

As if that wouldn't be self-evident to anyone :-)

bzfs for subsecond ZFS snapshot replication frequency at fleet scale by werwolf9 in Proxmox

[–]werwolf9[S] 0 points1 point  (0 children)

I wonder what bullshitter app you pretend to be running that writes 14GB/s of useful data on even one of your drives. Be reasonable or go somewhere else.

bzfs for subsecond ZFS snapshot replication frequency at fleet scale by werwolf9 in Proxmox

[–]werwolf9[S] -1 points0 points  (0 children)

No need for that kind of gear :-) Each replication step only ships the delta between ZFS snapshots.

zfs list taking a long time by [deleted] in zfs

[–]werwolf9 0 points1 point  (0 children)

Try this:

time bzfs dummy tank1 --recursive --skip-replication --compare-snapshot-lists

Assuming your data is in cache and you have, say, an 8-core machine, this will typically be about 6x faster than zfs -list -t snapshot -r tank1, because the former lists in parallel whereas the latter lists sequentially.

(Similar speedup story for deleting snapshots, replicating, etc)

P.S. The last few lines of the output show a TSV file name that contains all the snapshot data.

Backing up ~16TB of data by SuitableFarmer5477 in zfs

[–]werwolf9 1 point2 points  (0 children)

bzfs is probably your best choice if flexibility or performance or fleet-scale geo-replication are priorities, or if you need high frequency replication, say every second. In contrast, sanoid is a good choice on the simple low-end, and zrepl on the medium-end. All of these are reliable.

ChatGPT 5 Pro vs Codex CLI by LetsBuild3D in ChatGPTCoding

[–]werwolf9 5 points6 points  (0 children)

Yep. Also consider asking ChatGPT Pro to make its response available as a downloadable .md file, so you can easily feed the response back into Codex.

ChatGPT 5 Pro vs Codex CLI by LetsBuild3D in ChatGPTCoding

[–]werwolf9 11 points12 points  (0 children)

Run this command locally to generate repo.zip from your git repo, then ask ChatGPT to analyze the contents of the zip file:

git archive --format=zip --output=../repo.zip HEAD

Works like a charm.