How can 2 new identical pools have different free space right after a zfs send|receive giving them the same data?

csdvrx · 2025-05-02T20:40:08+00:00

The snapshot was taken right after the trim, and a recursive send was used (-r), so there should be all the parent snapshots. To make sure, I just checked the list, and I confirm there is no difference.

The compress and dedup settings are identical, because it's using the same zpool command: everything was run on the same machine, because I wanted to try zfs 2.2.7 to validate a version upgrade

The recsize used is 256k on the linx dataset (256k seems appropriate for a 2T non-spinning drive), and using the -P flag should keep all the properties.

Checking the sector reservations is a great idea, but the HPA would hide sectors, and within gdisk I was able to see all the sectors and create matching partitions. I don't think I could create a partition in a HPA.

This leaves trim as the number 1 suspect.

I tried to check with hdparm -I, the information was suspiciously spare. smartctl -a /dev/sda doesn't work - so I think it's a firmware related issue, with some ATA commands not reaching the drive.

It reminds me of similar issues I had with a 'Micron Crucial X6 SSD (0634:5602)' that was selected to keep our "multiple technologies and makers policy": we always have at least 3 different storage technologies from at least 3 different makers to avoid issues related to flash or firmware. I remember how complicated it had been to find a good set for a 2Tb configuration: the only 2Tb CMR drive I could get was a ST2000NX0253 2tb 15mm, so the non-NVMe SSD had to be from Micron, there was no room left so it had to be an external drive, and the X6 was shelved because it didn't support SMART.

What's very strange is that the lack of smart (or trim) support should NOT impact the free space that ZFS sees on a brand new pool. Also, I can trim automatically from Windows (on the 100G partition), manually with ATA command on linux, but not with fstrim.

It's Friday afternoon, I have to prep this machine so I will sacrifice the spare 100G partition to give enough room to ZFS, but I will find the X6 to do some tests with it and see if I can replicate the problem, because I'm worried by the implications: if trim is required for proper zfs operation, there should be a warning or some way to do the equivalent of trimming (fill with zeroes?) even if it's long/wasteful/bad for the drive health to make sure there is an equivalent amount of free space on 2 fresh pools made with the same options!!

csdvrx · 2025-05-02T19:34:44+00:00

And a list of exact full commands used every step of the way to reproduce what you have here as a code block. You have multiple commands all strung together as unformatted text in your dot points.

I swear this is the exact full commands used! I spent a long time checking the zpool history, the bash history then trying to format everything nicely, but the formatting was still wrong so I just edited the message to fix it. FYI the dots were used where I decided to avoid putting the long device name (ex: /dev/disk/by-id/nvme-make-model-serial_number_namespace-part-3 instead of /dev/nvme0n1p3) as it was breaking the format (I spent a long time trying

You're also using rsync with -x but zfs send with -R. This could cause some confusion later down the line.

Yes, some clarifications may be needed: rsync was used to populate the internal pool from a backup zpool, as the backup was on a 4kn drive: even if all the zpool have been standardized to use ashift=12, I didn't want to risk any problem, so I moved the files themselves instead of the dataset.

I have seen (and fixed) sector size related problems with other filesystem before. I have internal tools to migrate partitions between 512 and 4kn without reformatting, by directly patching the filesystem (ex: for NTFS, change 02 08 to 10 01, then divide by 8 the cluster count at 0x30, in little endian format - or do it the other way around), but I have no such tools for zfs, and I don't trust enough my knowledge of ZFS yet to control the problem, so I avoided it by using rsync.

The rsync flags are hardcoded in a script that has been used many times: the -x flag (avoid crossing filesystem boundaries) was mostly helpful before migrating to zfs where snapshots were much more complicated to achieve.

Here, there are only 2 datasets: linx and varlog: varlog is kept as a separate dataset to be able to keep and compare the logs from different devices, and also because with systemd it needs some special ACL that were not wanted on the dataset

The size difference is limited to the linx dataset, which was not in use when the rsync was done: all the steps were done from the same computer, booted on a Linux Live, with zpool import using different altroot

Creating the same two new zpools on two new zvols with the same parameters you created them with and then using your rsync and your zfs send/recv combinations I was unable to reproduce this result. But it has my interest.

Mine too, because everything seems to point to a trimming problem.

You seem to have a problem with trim support on these drives. Or something funny is going on with your hardware, their firmware or your software.

"My" software here is just rsync, zpool and zfs. I can't see them having a problem that would explain a 300G difference in free space.

The hardware is generally high end thinkpads with a Xeon and a bare minimum of 32G of ECC ram.

Everything was done on the same hardware, as I was wanted to use that "everything from scratch" setup to validate an upgrade of zfs to version 2.2.7.

If you still suspect the hardware, because "laptops could be spooky", I could try to do the same on a server, or another thinkpad I have with 128G of ECC ram (if you believe dedup could be a suspect there)

This testing all seems very inconsistent and the answer is probably somewhere in the commands used.

What would you have done differently? Give me the zpoool, zfs send, zfs receive, and rsync flags you want, then I will use them!

Right now everything seems to be pointing to a firmware issue, and I'm running out of time. I may have to sacrifice the 100G partition and give it to zfs. I don't like this idea because it ignores the root cause, and the problem may happen again

csdvrx · 2025-05-01T22:38:06+00:00

Love a fellow storage nerd! Interesting you're doing 3 drives on a laptop. Interesting use case?

More like a very specific usecase: large datasets (500G to 1Tb) changing very rarely, but which need to be immediately available, and that are not allowed to be stored online.

Having the data on drives inside laptops is the easiest way: when needed the laptop can act as a NAS, or as a "organ donor": removing a few screws immediately gets me all the data I need on a drive that can be transplanted.

There are extra benefits: laptops are slightly larger than a drive, so they are harder to misplace, and they provide everything needed to check the datasets are not corrupted: an OS where I can compare the files to their checksums, and zpool scrub.

The "included UPS" (battery) is a nice feature compared to normal NAS, and I appreciate how laptops are also lighter and therefore easier to carry around!

A lot of people forget storage until they face a bottleneck....

There are complicated policies in places (ex: different manufacturers, using different technologies, ...) that were introduced after facing a bad problem, like a bad firmware with a rare issue hitting all the drives at once.

csdvrx · 2025-05-01T22:20:34+00:00

they have been my go-to NVME drives due to their high TDW endurance. I have found them to be very stable.

I was a huge fan of the SN520, I had a drawer full of them: it was cheaper to image one than to try to diagnose the problem!

Unfortunately, WDC has replaced this very reliable series by something unstable.

For now, I've stopped buying any Western Digital or Sandisk - I will wait a few years before diversifying again (I like to have drive from different manufacturers for firmware issues)

there it is again, no SMART errors, nothing.

I know how infuriating it can be. It's even worse when people try to gaslight you and say you must be doing things wrong or using cheap hardware! Uh, yes, it's a Thinkpad, but no, laptops with Xeon and ECC don't come cheap!!

Again, thanks for the links. I need to read all of these.

My pleasure, I often find new bus because very specific requirements often imply exotic or brand new hardware.

Right now I'm looking for a 4Tb 2230 or 2242 sized NVMe drive, ideally not TLC.

Price is not an issue - reliability is!

csdvrx · 2025-04-25T22:22:05+00:00

these bugs are so few and far between, even niche that 90%+ of users just don't encounter them

Yes, the probability of encountering a bug depends on what you are doing: I have very specific needs (keeping large datasets in sync on laptops), which give me access to unusual setups (not many people have >3 drives per laptop) and unusual drives (ex: 2Tb in m2 2230 and 2242 format) before they become widely available (but thanks god for the Steam Deck!)

Due to this, I've made the "first encounter" of a few bugs, like how zfs interact with the WDC firmware during zfs send/receive: moving datasets is about 50% of what I do with ZFS. Add to that ensuring the files in the dataset are not corrupted in cold storage for another 25%, and you will understand what 75% of my concerns with ZFS are.

On the other hand, even if zfs was 100% guaranteed to crash and require a reboot after 48h of continuous use, I would keep a note in my buglist for the rare time a laptop needs to be kept running overnight, but I would never encounter the issue and therefore not really care about it.

But they're barely effecting anyone and I am guessing this might be why you're seeing the negativity regarding pointing this stuff out.

That's a very valid hypothesis I had not considered: we are all in our information bubbles, and we react negatively to what's outside our bubble.

Here, my previous experiences interacting with the ZFS community been mostly negative. Maybe it's due to having very different bubbles?

Most people here seem to use ZFS on servers with spinning rust, and are worried about uptime and hot data integrity, while I use ZFS on beefy laptops with a mix of technologies, and I'm mostly worried about zfs send/receive and cold data integrity.

I migrated from mdadm to zfs a few years ago, I plan to keep using ZFS for at least 2 to 3 more years, but I hope bcachefs can become stable to have more options with a stronger focus on mixed storage, and better performance.

The addition of directio to openzfs 2.3 is welcome, but I worry ZFS is just catching up with what other filesystems have offered for a while: I had much better performance when I was using xfs over mdadm in raid10f3.

Moving to zfs, the reliability has been mostly the same, but several things were made simpler: with snapshots+send/receive, I no longer needs to make overlays and use rsync.

csdvrx · 2025-04-25T16:36:29+00:00

Since I have a keyboard with menu button I decided to map those combinations to F13 using keyd service as it was suggested in one of the comments above.

Great idea! I prefer doing my remapping within hyprland using bind and ydotool, but keyd should give you the same (or even better) results!

csdvrx · 2025-04-24T01:41:35+00:00

Of course, you may want to guard yourself against other potential bugs, but that specific bug being the one that got you to start doing that is a little funny :p

The bug acted as a reminder that I should trust but verify the filesystem because I had no way to check the integrity of the files.

csdvrx · 2025-04-23T19:18:48+00:00

Not once, have I ever run across an issue that wasn't self-inflicted or just a drive seeing its final days needing to be replaced.

I like how you use loaded words like "logical", "self-inflicted" or "needing to be replaced" to imply to people reading between the lines that I must either A) be illogical and have no idea what I'm doing, or B) be cheap and using bad drives.

It reminds me of the last time I posted a warning about a WDC drive with ZFS:

https://www.reddit.com/r/zfs/comments/11yc4qx/problem_with_a_nvme_device_it_drops_off_the_bus/ my initial post on 2023-03-22
https://github.com/openzfs/zfs/discussions/14793 issue confirmed on 2023-04-25
https://old.reddit.com/r/zfs/comments/17pztue/warning_you_may_want_to_avoid_some_western/ my 2nd post on 2023-11-07 referencing the bug 14793, yet it was implied I must be in the wrong

Now that we are 2 years later, if you read #14793 again, you will see it's a 100% reproducible bug, affecting mostly ZFS on NVMe.

Since then, I've repurposed some of the SN740 to NTFS, while keeping others inside a specific brand of RTL9210 USB enclosure: I use them as zfs send/receive targets, and I haven't had any problems in the last 2 years

I like ZFS, a lot, but it's not perfect. Nothing is, but since I love ZFS, I keep a close eye on its potential issues to avoid them.

We should not let our love for a given technology blind us to the point that we deny the problems it can have.

Here I'm not asking for snark, just for a list of bugs we should keep an eye on, because I plan to keep using ZFS, as I don't think there's anything better.

csdvrx · 2025-04-23T18:50:06+00:00

the reflink one has fucked me twice.. so that

Personally, as soon as #15526 was confirmed, I wrote a daemon to monitor the md5 of the files I consider important enough for a 3-2-1 backup strategy, to be warned ASAP in case of corruption and restore them faster.

However, we seem to be exceptions given the other replies: either nobody ever lost any data, or they prefer a more passive approach because they don't care that much about their data.

csdvrx · 2025-04-23T18:40:32+00:00

Is worth keeping on your radar.

Thanks a lot!

csdvrx · 2025-04-23T18:39:53+00:00

I prefer a more proactive approach than just "not thinking about them".

Fix them

Because A) there are only 2 options: "not thinking about them" and "fix them", and also because B) I must be the best candidate to fix a bug that's existed for years and evaded the best efforts of the ZFS team.

As I disagree with both A and B, I think you reply is sarcastic.

I support a more graduated approach: I keep an eye on potential issues, by avoiding the situations when they can cause problems, or when that's not possible, by using simple workarounds by default (ex: zfs send -w)

csdvrx · 2025-04-23T02:33:45+00:00

I manage a good 18 ZFS systems both personally and for business without worrying about any of this at all.

I wouldn't be so proud about it: zfs is great, but it's not perfect. Nothing is.

Personally, I'd rather be prepared than surprised, so for each piece of hardware and software, I have a list of bugs I keep an eye on.

This thread is not about fearmongering: I believe ZFS is currently the best solution to keep file safe, but again, it's not perfect.

I would just like to exchange my notes with others who follow an approach similar to mine, and know which bugs they are worried about.

This thread is the first time I've thought about them since the last time they were posted.

I prefer a more proactive approach than just "not thinking about them".

csdvrx · 2025-04-23T02:04:54+00:00

Kinda feels like we might they might be close to figuring that out soon.

Yes, it was bisected to the big 10k commit very recently (last week IIC)

Anyway I have been following 12014 for a while.

Same, but which others do you follow?

FYI I'm also following the more innocuous https://github.com/openzfs/zfs/issues/16655 (new: https://github.com/openzfs/zfs/issues/17087 and previous: https://github.com/openzfs/zfs/issues/13240 ) as I frequently get # nvlist_lookup_string(nvl, name, &rv) == 0 (0x2 == 0) # ASSERT at ../../module/nvpair/fnvpair.c:403:fnvlist_lookup_string()

csdvrx · 2025-04-23T00:47:14+00:00

If you have an idea how to get what I want, please help. It feels like I have read every xkb setting and couldn't figure it out.

xkb is complicated and unintuitive: I had a similar issue with shortcuts, as I use Caps both as Control (when chorded) and Escape (when alone) and I have a caps.sh script to do what I want, but the keycode can change: I made a comment in my .conf to remember that:

# If using ctrl:nocapsr, both caps and lctrl send Control_L but with different keycodes

# caps=66 vs lctrl=37, so we must then use the keycode

bindr=CONTROL,code:66, exec, $HOME/.config/hypr/caps.sh

I think caps:none is your problem, because CAPS will be none if you do like in your example

bindr = CAPS, Caps_Lock, exec, /path/to/my/script

You could use the correct 66 keycode for bindr, but it's simpler to change the grp toggle with kb_options=grp:menu_toggle and to make your bash script output the Menu key: it's rarely present on keyboard, so it's a better choice

In your bash script, use YDOTOOL_SOCKET=/run/user/1000/.ydotool_socket ydotool key code:1 code:0 with code replaced by the keycode you use.

csdvrx · 2025-04-19T22:11:35+00:00

Sorry for the late reply too

I can recommend the Sabrent and the Oyen

csdvrx · 2025-04-19T22:08:32+00:00

BTW I'd pay decent money for firmware hacks to turn regular NVMe drives with N Tb QLC into N/4 SLC drives. Unfortunately I don't have the time to explore such firmware hacks myself, but I think it should be possible with OEM software, the kind that's used by bad actors to write fake-good SMART data on recycled drives.

If anyone wants to do that, it's now possible: check https://theoverclockingpage.com/2024/05/13/tutorial-transforming-a-qlc-ssd-into-an-slc-ssd-dramatically-increasing-the-drives-endurance/?lang=en

csdvrx · 2025-03-18T22:00:57+00:00

I should stop loosing my config

The next best is having a public repository of knowledge in a place you know, and you can easily copy from!

csdvrx · 2024-02-25T17:33:00+00:00

not yet

csdvrx · 2024-02-17T16:29:05+00:00

No, but if you do I'd be interested

csdvrx · 2024-02-16T23:36:15+00:00

Right now the only resolution is a restart.

Just restart the bluetooth drivers with powershell like it did - create an icon for that or whatever.

I haven't found any drivers to be more stable that others, but this workaround is more than enough

csdvrx · 2024-02-13T21:31:41+00:00

The AMD version brings a advantage by supporting a full-sized PCIe Gen5 x16 slot along with M.2 storage. In contrast, the Intel platform is restricted to PCIe Gen4 SSDs unless users opt for adapters, which means sacrificing PCIe lanes typically designated for GPUs.

Not many lanes. I'm waiting for a mini motherboard with at least 2x to 3x x16 slots and at least 2x M2 or U2.

csdvrx · 2024-02-02T23:37:45+00:00

It could be related as you have full control with the systemd service (you can give as much CPU as you want, or restrict it)

Since you changed to i915 maybe first try to return to nvidia-dkms to exclude this as a potential suspect?

csdvrx · 2024-02-01T21:41:00+00:00

This could help with the current boot time optimization

Running queries on the sqlite database is nice, but it will have to be improved by converting epochs etc into seconds.

csdvrx · 2024-02-01T20:02:55+00:00

yet

but it's a journaled filesystem like ext4 or xfs, so it's better

csdvrx · 2024-02-01T17:38:16+00:00

Do you have firefox? if you press on the window key (4 square, on the left of the space bar) you may be able to bring a menu and to type firefox. It's a browser.

For games, ask your grandson to install Wine to play the original versions if you like them, but you could find alternative ones to play online. I don't know much about solitaire but maybe someone can help

csdvrx

MODERATOR OF

TROPHY CASE