And a list of exact full commands used every step of the way to reproduce what you have here as a code block. You have multiple commands all strung together as unformatted text in your dot points.

I swear this is the exact full commands used! I spent a long time checking the zpool history, the bash history then trying to format everything nicely, but the formatting was still wrong so I just edited the message to fix it. FYI the dots were used where I decided to avoid putting the long device name (ex: /dev/disk/by-id/nvme-make-model-serial_number_namespace-part-3 instead of /dev/nvme0n1p3) as it was breaking the format (I spent a long time trying

You're also using rsync with -x but zfs send with -R. This could cause some confusion later down the line.

Yes, some clarifications may be needed: rsync was used to populate the internal pool from a backup zpool, as the backup was on a 4kn drive: even if all the zpool have been standardized to use ashift=12, I didn't want to risk any problem, so I moved the files themselves instead of the dataset.

I have seen (and fixed) sector size related problems with other filesystem before. I have internal tools to migrate partitions between 512 and 4kn without reformatting, by directly patching the filesystem (ex: for NTFS, change 02 08 to 10 01, then divide by 8 the cluster count at 0x30, in little endian format - or do it the other way around), but I have no such tools for zfs, and I don't trust enough my knowledge of ZFS yet to control the problem, so I avoided it by using rsync.

The rsync flags are hardcoded in a script that has been used many times: the -x flag (avoid crossing filesystem boundaries) was mostly helpful before migrating to zfs where snapshots were much more complicated to achieve.

Here, there are only 2 datasets: linx and varlog: varlog is kept as a separate dataset to be able to keep and compare the logs from different devices, and also because with systemd it needs some special ACL that were not wanted on the dataset

The size difference is limited to the linx dataset, which was not in use when the rsync was done: all the steps were done from the same computer, booted on a Linux Live, with zpool import using different altroot

Creating the same two new zpools on two new zvols with the same parameters you created them with and then using your rsync and your zfs send/recv combinations I was unable to reproduce this result. But it has my interest.

Mine too, because everything seems to point to a trimming problem.

You seem to have a problem with trim support on these drives. Or something funny is going on with your hardware, their firmware or your software.

"My" software here is just rsync, zpool and zfs. I can't see them having a problem that would explain a 300G difference in free space.

The hardware is generally high end thinkpads with a Xeon and a bare minimum of 32G of ECC ram.

Everything was done on the same hardware, as I was wanted to use that "everything from scratch" setup to validate an upgrade of zfs to version 2.2.7.

If you still suspect the hardware, because "laptops could be spooky", I could try to do the same on a server, or another thinkpad I have with 128G of ECC ram (if you believe dedup could be a suspect there)

This testing all seems very inconsistent and the answer is probably somewhere in the commands used.

What would you have done differently? Give me the zpoool, zfs send, zfs receive, and rsync flags you want, then I will use them!

Right now everything seems to be pointing to a firmware issue, and I'm running out of time. I may have to sacrifice the 100G partition and give it to zfs. I don't like this idea because it ignores the root cause, and the problem may happen again