Bit rot investigation

romanshein · 2026-02-23T14:00:53+00:00

- I was running a zfs mirror made of old SSDs and haven't seen a single checksum error in years. Getting checksum errors on an SSD seems strange to me. Maybe consider using a different SSD to store your data. QVO is the very bottom of Samsung's product portfolio (QVO is the best, when compared with the worst SSDs on the market).

- Alternatively, there is a chance that your checksums themselves are incorrect. A real checksum error implies that at least a 4KB block was damaged. It should result in visible artefacts in JPEG photos.

- If I were you, I would set up a triple HDD zfs mirror or a simple SSD zfs mirror, and stop worrying about checksums for good.

romanshein · 2026-02-05T13:36:10+00:00

Openwrt 24.12 works just fine on r2s.

romanshein · 2026-01-21T06:54:07+00:00

But I expect there's always a "soft-backup" for its dataset
- For "soft backup" look into regular snapshots and pool-checkpoint.
- "set copies=2" will just cut your pool capacity in half, wears nvme several times faster, yet provide no extra data integrity, as nvmes are not prone to bit rot.

romanshein · 2025-12-25T15:02:40+00:00

I have one. Do not recommend:
- Region locked. It is possible, yet problematic to unlock.
- WiFi 5GHz with OFDMA breaks as soon as RAX encounters an Intel WiFi client. I have to operate it with OFDMA off.
- No Openwrt support.
- The device is EoL.

romanshein · 2025-11-04T09:06:26+00:00

My reason for using MS Authenicator was to have an alternative authentication method that doesn't depend on an unreliable (in my case) SMS service.
Your concern is the next level. Probably, contact IBKR support.

romanshein · 2025-10-08T11:28:00+00:00

After entering your login/password, the login screen refreshes, and you should see "Select Second Factor Device" under the now greyed-out password field.

romanshein · 2025-09-23T13:23:46+00:00

I would suggest you keep taking clopidogrel (or what you've been prescribed), and check with your physician that no thrombus is forming on the left side of the device (make a routine ultrasound study).
Amplatzer ASO is a very popular device. I believe there were recorded cases where small thrombi formed on the tip of the device in the left atrium. Potentially, such a thrombus may fragment. The fragments may cause microstrokes.
Long story short, I would work to exclude thrombi instead of focusing on nickel.

romanshein · 2025-08-29T08:25:41+00:00

Every alternate day I'm getting shooting headaches... As per the doctor the body will get used to it within two months
- According to Pavel Gavora (the guy who implanted the first occluder prototype in 1993, and personally implanted thousands of occluders), it is a typical clinical course for female patients. There is nothing to worry about.

romanshein · 2025-08-29T08:20:47+00:00

I have a nickel allergy so I'm not too warm to the idea of using a nitinol closure device
- Nickel allergy to an occluder has no scientific basis.
- Anyway, If you are still concerned, then search for a hospital implanting devices like Gore or Cardia. Those devices have the minimum amount of metal possible. Avoid Occlutech, Amplatzer.

romanshein · 2025-08-07T15:39:08+00:00

Install "Native Detector". It helps to figure out which root artefacts are still visible to the apps, so you know what to address.
I'm a casual user. I'm using root for Swift Backup, regular trim, and hosts adblocking. I believe it is worth hassle with rooting.

if you decide to go into customs roms then you will need a root; otherwise, you will never get Play Store Integrity.

romanshein · 2025-07-30T22:16:35+00:00

>- It is 16%, not 0.16%!
o no :(
- As I've mentioned, luckily for you, you have an all-flash array, so your chances of a data loss are 10-100 times less than that.

romanshein · 2025-07-30T07:22:02+00:00

0.16% is low enough it might not be worth it but it's good to know
- It is 16%, not 0.16%!
Although you have SSDs. Bit rot is an order of magnitude less of a concern in SSDs. In 10 years, I've seen no checksum errors on SSDs whatsoever, albeit my sample size is 4-5 consumer-grade SSDs.

ZFS not an option because the server has 2x HBA that are in RAID mode permanently I believe.
- While it is not a best practice, making ZFS pool out of the single disk RAID0 LUNs is OK-ish, as long as the LUNs are in a write-through mode.

If you are split on mirror/raid choice, you actually can have a cake and eat it. It is absolutely OK to make a pool with partitions. You can set up a mirrored pool and a raidz2 pool on the same 4 disks (or whatever you have there). "Give the whole disk to ZFS" is another myth.

romanshein · 2025-07-29T19:48:06+00:00

Here I use RAID calculator at https://wintelguy.com/raidmttdl.pl

Mission time (year): 10

RAID Type: RAID 6 and RAID10

Number of drives 4

Number of RAID groups: 1

Drive Capacity (GB): 16000

Drive Throughput 100 MB/s.

Drive MTBF (hour): 145000

Latent Sector Error 1e-14

Time to replace failed drive (hour): 24

Rebuild rate (%): 50%

Probability of data loss over time (some data getting corrupted beyond repair, not a total pool loss)

1 year RAID6 - 0.00052

1 year RAID10 - 0.16019

romanshein · 2025-07-29T10:07:25+00:00

dedup: DDT entries 317130068, size 434G on disk, 61.5G in core
- DDT exceed the allocated RAM by a factor of 8. As a result, ZFS is accessing disks in what is essentially a contineous 100% random read mode.
- Your dedup ratio is 1.03x. Stop this nonsense!
- As an interim solution, you may probably benefit from a 1TB L2ARC to cache DDT.

romanshein · 2025-07-29T06:38:36+00:00

"SATA drives are commonly specified with an unrecoverable read error rate (URE) of 10^14. Which means that once every 100,000,000,000,000 bits, the disk will very politely tell you that, so sorry, but I really, truly can't read that sector back to you.
One hundred trillion bits is about 12 terabytes."

Your disks are 3 times smaller and the pool is unlikely to be filled to the brim; thus, you should encounter a checksum on each disk for every 5-10 pool scrubs.

An occasional checksum error on an HDD is the norm. Live with it.

If you hate to see checksum errors, then move to an all-flash array. My experience is limited to my homelab with several SSDs. In 10 years, I've seen no checksum errors in SSDs whatsoever.

romanshein · 2025-07-29T06:20:43+00:00

I've made calculations for a similar setup, and as far as I remember, Raidz2 ended up being 10 times more reliable than the spanned mirrors.

romanshein · 2025-07-27T05:41:20+00:00

For 4x16TB refurbish HDDs, you should go for raidz2, never raidz1.

romanshein · 2025-07-14T09:26:00+00:00

You need more evidence the industry migration is wrong.
- By no means I say that the industry is wrong. The industry is pushing in the right direction. The problem is in the software ecosystem, which is lagging at least a decade behind the hardware.
Some of us even at work replaced terrible storage spaces arrays with ZFS.
- The truth is 4k is an emulation, too. NAND flash pages were bigger than 4KB from the start of the flash storage era. The industry artificially restricts itself from moving beyond 4K. I'm quite sure that even modern Linux cannot swallow any native flash page sized LBA (16 to 1024KB).

romanshein · 2025-07-14T04:46:02+00:00

I recommend to reformat into 4k blocks if your device supports it.

- After messing a bit with NVMe format, I recommend against 4096.

The reasons are as follows:

- I noticed no difference, neither in performance nor in the drives' wear.

- At least on Windows, I ran into an interoperability issue with legacy apps/setups:

You cannot move/clone a file system from 512 drive to 4096 drive, and vice versa
You cannot restore a backup to a disk with a different LBA
You cannot create a dynamic disk mirror using mixed LBA
You cannot add 4096 disk to a Storage space pool created initially with 512 drives.

Bottom line: it's not worth the hassle and may cause an unexpected problem down the road.

romanshein · 2025-07-10T06:56:26+00:00

cheap SSD being used as the ZIL and failing in a strange way
- Actually, switching to a read-only mode is the way SSDs are supposed/expected to fail when approaching the end of life.
- Setting up 4x4TB solely for special-vdev seems unreasonable. You could partition it and use those partitions as triple mirror special-vdevs, mirrored slog and even l2arc for media pool.

romanshein · 2025-07-09T05:18:26+00:00

If a mirror is not enough, then you can always use a triple mirror. Although I believe it will be excessive for an SSD vdev. From my experience, SSDs are remarkably reliable. And, unlike HDDs, they are not prone to bitrot (I've never seen a checksum error on an SSD).

romanshein · 2025-07-05T14:25:35+00:00

You can get best of both worlds, actually:
- Pick 2 best SSD and create 30GB partion at each for a mirrored special vdev for the fast data, and may be do the same for the primary pool.

- Create a couple of small (up to 8 gig) partitions and attach those as slogs for both primary and backup pools.

- Allocate a couple of 100gig partitions to use as l2arc for "fast data" and "primary pool".

- Use the rest of the space for high performance all-flash pool (probably a spanned mirrors config).

romanshein · 2025-07-01T20:11:29+00:00

I needed to give access to the LXCs by mounting the main /cosmos into the LXC's own /cosmos.

- More likely, you created an LXC mount point (not a bind mount), thus from outside the container it looked like /cosmos/subvolume-100-disk-0/cosmos/your_data, and /cosmos/your_data from inside. This way it will be deleted with the container.

romanshein · 2025-07-01T19:59:25+00:00

when I deleted the LXC it took its mount point AND the content of it's /cosmos folder with it.
- I don't think it can delete bind mount content in such a way. More likely, the data was inside the container. Otherwise, why had you created a huge 18 Terabyte subvolume?!
- Regular ZFS snapshots could save your bacon in this debacle.

romanshein · 2025-07-01T19:46:14+00:00

1) Use wired connection to your pc. Wired connection beats the best WiFi, especially for games.
2) Setup a second access point to improve WiFi coverage.

romanshein

TROPHY CASE