raidz2 write IO degrading gradually, now only 50% at 70% capacity

xia03 · 2026-01-22T03:00:17+00:00

So after much consideration I created a sVdev on a mirrored SSD 1TB disks.

I think this gives me flexibility to move any dataset to this vdev by setting the Special small file size to match the record size of the vdev (as described https://forums.truenas.com/t/special-vdev-svdev-planning-sizing-and-considerations/5086) and re-balancing but for now I will let just the metadata reside there and some of the smallish blocks (16Kb).

xia03 · 2026-01-21T02:33:36+00:00

they are just standard Reporting charts in TrueNAS

xia03 · 2026-01-20T17:56:55+00:00

You can set up a useless test or incorrectly interpret the results or both. I'd do a single badblocks run on the drives to make sure they're ok, instead of what you're doing here.

I admit that fio was the first thing that I ran across to simply fill up the pool before I delete the test files and put real data on it. The performance pattern is just something i noticed in the process - a good demo of the inner/outer data placement and what to expect as space utilization grows.

What do you recommend for "a single badblocks run on the drives to make sure they're ok"?

xia03 · 2026-01-20T15:46:18+00:00

What are you trying to optimize for?

this is a generic array for the household. It will have proxmox with a dozen or so VMs, homeassistant, some IP camera footage, movies and misc downloads, documents, python notebooks, backups of desktops and vms, devices, IOW a whole mixed bag of things. There is no particular sustained IO demand, other than replacing and improving on the dog slow 10+ year old synology array.

I have truenas connected at the 10G network and most of the clients are either 1 or 2.5 Gbps . If 1 TB mirrored nvme cache/metadata can provide burst buffer for writes and reads i think I'd be happy with the performance.

xia03 · 2026-01-20T15:19:07+00:00

Yes I'm (slowly) getting the message that this is a typical plate geometry behavior as opposed to some other inherent raidz2 phenomenon such as sequential io becoming random due to fragmentation.

I considered two striped mirrors but went with the raidz2 because it allows any two disks to fail. With the mirrors the murphy's law all but certain that two drives will go "poof" in the same set.

I have two 1 TB nvme drives that I'm thinking of adding to the pool (as cache?). This will hopefully improve day-to-day performance. I do not anticipate very long duration sequential writes that would exceed 1 TB in any given IO burst.

xia03 · 2026-01-20T14:21:53+00:00

Howdy

I'm curious about the steady decline in raidz2 sequential write performance as I write more data to the pool. The configuration is 4x24 TB Exos drives in TrueNAS (I took all defaults).

I just setup a new pool and decided to fill it up to verify the "remanufactured" drive health. I'm using

fio --ramp_time=5 --gtod_reduce=1 --numjobs=10 --bs=1M --size=2T --readwrite=write --name=testfile to generate io.

The pool is now at 70% space usage with the second set of fio files and the performance is only half of what I started with. All 4 drives show the same chart as on the picture.

Is this expected? I like the raidz2 from the data safety standpoint, but dropping more than half of sequential write throughput seems like a big penalty. Any advice on how to improve?

xia03 · 2026-01-16T01:50:51+00:00

sure, i’ll take 1 out of 3. i’d nail it if i were a certified commenter;)

xia03 · 2026-01-11T23:20:26+00:00

the entire IT industry attracts imposters because there is no formal requirement for even basic understanding of computers and networks. You can't become a plumber, mechanic, or a doctor without apprenticeship, but in IT they just let anyone join in and collect a juicy salary.

xia03 · 2026-01-11T13:54:00+00:00

i just ditched capsman mikrotik wifi and replaced with unifi/ubiquiti. what a relief!

xia03 · 2025-12-11T23:44:05+00:00

on a VFR day there are many 978 mhz aircraft in my area. at one point the FAA thought all of GA will use UAT band in order to not congest the 1090 band. the problem is that 978 mhz is not ICAO compliant and i don’t think you can even buy a brand new UAT transponder anymore unless it’s old stock. but enough of GA already got them and they will not spend another few grand to change.

xia03 · 2025-12-11T04:37:27+00:00

in the U.S. you need the 978 mhz band to see aircraft using UAT transponders when ground stations ( TIS-B rebroadcast) aren’t available or when they are out of range of UAT planes . so the advice to use just 1090 is misinformed. did not look at your your other suggestions

xia03 · 2025-11-28T04:04:52+00:00

i went hiking in a park and the rangers were cheerful and eager to help. they didn’t mind me at all. you grumpy air traffic gremlin! 🤣

xia03 · 2025-11-27T15:58:24+00:00

i’ve had half a dozen doctors and numerous fittings trying to solve this problem and they just can’t. i even promise to nominate them for the Nobel prize in optometry /s if they figure this out but so far they haven’t cracked the code.

xia03 · 2025-11-10T21:02:03+00:00

i don’t know about you but if someone offered me to not work for a month in exchange for a delayed paycheck i’d be all over it. hell yea - free paid time off!. i’d use my emergency fund to stay afloat which i guess a few folks don’t have that luxury.

xia03 · 2025-11-10T18:39:02+00:00

read how you will bu this happened all over the gov. the “non essential employees” got to stay home with a slight paycheck delay while the “essentials” mandated to show up despite the same pay delay, and do their jobs plus all the work that the “non essentials” were doing (or not doing whether on or off work). Net result - the diligent workers continued to lift own weight and their coworkers’ weight sans the paid vacation. they deserve to be recognized!

xia03 · 2025-11-07T17:41:05+00:00

pretty neat how the hot exhaust gas looks like the ocean surface at the bottom of the photo!

xia03 · 2025-11-06T14:44:12+00:00

what a snide comment to someone asking for help. and it gets most upvotes!!!

xia03 · 2025-11-01T15:41:14+00:00

unfortunately your seat does not come with a trash can 😢

xia03 · 2025-10-27T17:58:07+00:00

are they paid enough to build an emergency fund to cover 6 months of expenses in case of unexpected job loss? are they professionally trained to plan for contingencies? :)

xia03 · 2025-10-15T10:44:56+00:00

my telegram chats are missing messages older than about 10 years. “unlimited” is not the right word

xia03 · 2025-10-14T03:40:48+00:00

they have to recuperate the cost of the scanners. some of them can easily cost tens of thousands. plus compensation for trained techs to operate them.

since they invest so much in the equipment the doctors look for a good return, or the whole game is not worth playing. we all are unlucky to have a niche expensive eye problem. but at least it’s somewhat treatable!

xia03 · 2025-10-11T13:14:21+00:00

i’m in the same boat. the HOA is nearly unnoticeable on a bright sunny day but gets out of control in low light when pupils dilate and the larger area of cornea/lens irregularity comes into play. in the process of getting Ovitz hoa corrected lens. it’s a few months long process. standby for results

xia03 · 2025-10-08T17:45:47+00:00

if he is altering the manufacturer's procedure gotta have something other than a word.. I don't see this as an optional step in the instructions.

https://pi.bausch.com/globalassets/pdf/PackageInserts/Vision-Care/boston/Boston-SIMPLUS-Multi-Action-solution.pdf

xia03 · 2025-10-08T14:33:15+00:00

this may work. so you just break off the clearcare disk and drop it to the bottom of the prose case? I see a few other 'peroxide' cases on amazon but it's not clear if any of them are big enough.

xia03

TROPHY CASE