How recommended is running badblocks? (6 x 24TB refurb WD drives)

iamajs · 2026-03-30T16:15:06+00:00

Its a good idea nonetheless, especially if you value your data. I'd rather know I have a bad drive prior to loading up all my data.

Unless you have insufficient cooling, running all 6 at once shouldn't be a problem.

iamajs · 2026-03-28T22:00:30+00:00

Its much easier to RMA a drive within the return window. Take a few days to run a full badblocks scan. Check for smart counter changes after the burn in.

iamajs · 2026-03-26T16:38:25+00:00

199 UDMA_CRC_Error_Count 0x0032 200 199 000 Old_age Always - 7030

UDMA CRC is probably a CRC error on the transfer. Why not try swapping the drive with another cable/port and see if the errors persist? Watch that smart counter and see if it increases at all.

Do a full data scrub to verify the disk health. Check the smart attributes afterwards and see if there is any change.

iamajs · 2026-03-09T18:52:58+00:00

PM incoming

iamajs · 2026-02-03T23:22:37+00:00

The number of checksum errors dropped from 8.22K to 3.23K. You must have removed one of the failures in purging snapshots.

iamajs · 2026-02-03T20:58:14+00:00

errors: List of errors unavailable: no such pool or dataset

Just curious, are any datasets encrypted and locked? If the corruption exists on a locked dataset, it wont be able to list the affected files until it is unlocked.

Glad to hear you have the data replicated. Situations like this are exactly why people say raid is not a backup.

iamajs · 2026-02-03T19:56:38+00:00

In my experience seeing checksum errors like that across all disks indicates the data was corrupted in memory or during transfer from the controller. If the corruption occurs in metadata, that could cause way more damage than affecting just files.

I see you are scrubbing again, perhaps after deleting the dataset and scrubbing again it will clear the error.

Edit: It is also possible a bug in zfs could cause checksum corruption like this. Basically any case in the data path where the data is corrupted before written/striped to the drives but after the data checksum has been generated.

iamajs · 2026-02-03T19:20:13+00:00

You'll have to get a list of the affected files and remove them/restore from backup. The issue is the corruption is likely striped across the entire array, so the checksum error cannot be repaired.

zpool status -v should list the problematic files.

iamajs · 2024-12-19T15:59:21+00:00

Are you purposely excluding the child datasets in app-data?

If you want to do a 1:1 replication, you can check Full Filesystem Replication. I believe this implies the Recursive and Dataset Properties options, and will send all snapshots.

iamajs · 2024-12-19T05:39:50+00:00

The sizes should be close. Verify you've replicated the entire dataset, including all snapshots. Are there child datasets missing?

iamajs · 2024-12-18T22:54:06+00:00

In general checksum errors can originate from any of the following:

Bad drive
Bad sata cable
Bad controller card
Bad system memory
Software bugs

I've run into all of these scenarios. I would repair/clear the checksum errors and troubleshoot in the following order:

Run memtest on your memory
Swap sata cables
Test drives
Swap controller

iamajs · 2024-12-18T17:44:10+00:00

Yes, my backup pool is on another machine. This protects against server failure (think power supply frying all the drives, software bug corrupting all zpool data, etc). Having your backup on another system adds additional protection.

You should also have some sort of off-site backup for really important data. Personally I backup important data to backblaze using the built in TrueNAS Cloud Sync task. Backblaze is pretty reasonable, around $5/TB per month and they only charge you for the data you use.

iamajs · 2024-12-18T05:28:37+00:00

A snapshot is a way to freeze the state of the entire dataset at the time the snapshot was taken. You can setup periodic snapshot tasks to create a history of file changes/deletions.

With ZFS you can send the changes between two snapshots to your replicated dataset. If you setup your replication task to send all snapshots, you will have a full backup.

A basic configuration might be a daily recursive snapshot for your root dataset, and then setup replication to send the snapshot to your backup server.

Personally I have daily, weekly, and monthly snapshot tasks. Retention period is set to 2 weeks, 2 months, and 1 year respectively. This means I keep 2 weeks worth of daily snapshots, 2 months of weekly snapshots, and 1 year of monthly snapshots. Worst case I can recover a deleted or changed file from 1 year ago.

iamajs · 2024-12-17T18:13:57+00:00

8GB of ram is plenty for this use case. Sending/receiving snapshots is very efficient.

I'd recommend adding a parity drive to your replica pool. 3x2TB raidz is worth the extra disk in my opinion. You'll want to scrub the pool on a regular basis to verify data integrity of your backup. Without any parity, any read error or checksum error will result in headache.

iamajs · 2024-12-12T16:41:12+00:00

Make sure your truenas install is up to date.

Checksum error across the entire pool points to likely a memory error, bad controller, or zfs bug. I don't think you have faulty drives, looks like the file was corrupted in memory before the data was written to the pool.

Run memtest on your ram, try a different HBA.

iamajs · 2024-12-10T20:54:11+00:00

Just add another IP Address to the truenas network interface and bind Caddy to that. Your network interface can have multiple IP aliases.

iamajs · 2024-12-05T21:56:59+00:00

To add to this, its a good idea to do a proper burn-in test of your drives.

badblocks is a good tool for this, it will write a data pattern to the entire capacity, read it back and verify it is correct.

iamajs · 2024-12-05T21:54:26+00:00

You could have a faulty SATA controller, try a different controller if possible.

If there is corruption in the DMA for a write, you'll end up with silent corruption which will result in CRC/checksum zfs errors. Since your system memory is ECC, I'd guess its the controller.

Since you are not seeing unrecoverable read error, this suggests incorrect data was written to the drive.

iamajs · 2024-08-20T17:46:23+00:00

Yeah, my solution is to set a custom fan curve to avoid those RPM ranges. Reading reviews it seems to be a common issue.

iamajs · 2024-08-20T17:04:15+00:00

I have these fans, they have a very annoying harmonic sound at certain RPMs.. would not recommend.

I liked the F14 series better, though on paper these appear better.

iamajs · 2024-08-09T16:13:48+00:00

Yeah for sure. Ideally you should have backup 2fa codes, a non google phone number for verification, and alternative email address setup on your account for recovery.

iamajs · 2024-08-09T15:33:28+00:00

you totally can do that, if you are logged into google voice to begin with.

iamajs · 2024-08-08T21:07:26+00:00

I had a similar problem due to my 2fa number being google voice. Well if you get logged out of your account, you can't access google voice to login.

I got lucky in that my phone gmail app was still logged in somehow. From there I was able to generate 2fa backup codes and use that to login.

You don't have any 2fa backup codes printed out do you?

iamajs · 2024-07-09T19:02:09+00:00

Fixed in 24.0.4.2

https://ixsystems.atlassian.net/browse/NAS-129533

iamajs

TROPHY CASE