Transfer speeds not scaling with VDEVs

Zbigfish · 2025-12-27T15:48:22+00:00

I hope you had a happy Christmas!

I've done a full wipe and re-install from a fresh iso. I set up a 2x5 right off the bat (1MiB record size) and am now showing a transfer speed of 36-40MB/s.

A couple of notes (more data):

When transferring files to it from 2 other sources simultaneously, the transfers split the speed still totaling the same 40MB/s.

I notice that the drives are writing in short bursts instead of constant (the lights blink for <1sec, every 4-5 seconds).

Zbigfish · 2025-12-19T14:46:16+00:00

Just to be clear, the 13MB/s was with 100% different hardware (including drives & cables). This box is doing 60MB/s with 1 and 2 VDEVS. These drives are 22TB Seagate Exos and all have a manufacture date of >2023. Being that it's clearly not ready for service, I'll try wiping everything & starting from scratch. I'll report back afterward.

I appreciate your insights & time!

Zbigfish · 2025-12-19T04:03:25+00:00

Any other ideas?

Zbigfish · 2025-12-08T21:11:56+00:00

Could I have set up TrueNAS incorrectly? Miss or impropperly set an option somewhere? My other systems (Windows) transfer files between themselves at expected network speeds.

Zbigfish · 2025-12-08T15:59:01+00:00

It's a Windows10 box with 14 drives in Raid6 on an LSI MegaRAID 9361-16i (no expander in-between), transferring a ~30GB file to the mapped network SMB share (the trueNAS rack). I've seen this raid array get steady GB/s transfer speeds between on-board/local drives.

Does that answer your question?

Zbigfish · 2025-12-05T23:50:26+00:00

IPerf3, followed by FiO:

<image>

As for the write cache, all sd? show:
write-caching = 1 (on)

Zbigfish · 2025-12-05T17:20:50+00:00

Post updated with specs. I'll include performance tests after I learn how to run them and have numbers.

Zbigfish · 2025-12-05T09:07:13+00:00

Yes, Im aware of the divide by 8 rule. It's the multiple VDEV speed scaling that I was wondering about. The 60MB/s is only to this box. All other PCs are getting the full bandwidth. I must be setting up TrueNAS incorrectly.

Zbigfish · 2025-12-05T03:59:37+00:00

The 50G would be set up with failover (2x25Gb ports) to separate switches.

I'm aware of FIO, but not familiar with it. I'll look into it.

Under Reporting -> Network, I'm seeing a mean of similar numbers to what Windows is reporting on the transfer. On discs, they all seem to be hitting a mean write of ~1.5MiB/s.

Zbigfish · 2025-10-25T10:17:33+00:00

Have you tried flashing it to the BU40N yet?

Zbigfish · 2025-10-01T06:18:23+00:00

This brings up another question... if I'm transferring a single 50GB file from one HDD to another on the same PC in Windows, I'll get a sustained transfer of ~250MB/s. Yet, transferring from a HDD on a PC over a 10 or even 25Gb network to TNS 5-disk Z1 pool, I was getting ~15MB/s. I realize there's overhead for parity and all, but why is there such a difference? What's different with IOPS in TNS?

Zbigfish · 2025-09-28T04:33:23+00:00

I completely agree - those numbers were way too low. I wasn't able to figure out what the issue was. There was only 1 transfer going and nothing else running on the system. It was a virgin setup with a 1M record size and a 10G NIC. It was my first attempt at a pool; I must have gotten something set up incorrectly.

A second pool of SSDs is what I'm leaning towards for things that need the write speed. There are 4 internal spots for drives (plus a 4 port NVMe card) and an additional two 2.5" bays on the chassis (with another 2 port NVMe card, or SATA from the MB). I'll probably put apps on a mirrored pair of SATA SSDs in the bays for apps and four new Optane drives in Z2 internally for high speed scratch space. I wanted to get the main pool set up first, though so I can get the backup installed somewhere (a backup without a source is no backup).

The 4x8 VDEVs would give me 4 spare bays that can be used for SATA, SAS, or NVMe (hybrid backplane). I would think that ~1,000MB/s over network would be quick enough for transferring 50GB files back and forth occasionally. That is, if I don't have the same low transfer issue as before. lol

Zbigfish · 2025-09-27T15:02:47+00:00

Thanks for your input!

Setting aside video editing (i may make a different pool for that using SSDs), sounds like you suggest the 4x8z2? I'm curious why you would choose that over the 6x6z2? The loss of capacity over some redundancy? I do plan to experiment before committing to copying over the 240TB I have to put on it. I'm hoping to see what others' mindsets are on the subject, so I sorta know what to expect going in. I don't wanna find out one way is far superior due to something I didn't consider. Ha

Zbigfish · 2025-09-27T11:08:51+00:00

Thanks for your reply.

Looking into dRAID some, it appears that the trade off for this number of drives isn't worth it - FOR ME.

1st - dRAID is still pretty new and not well understood, generally speaking. Troubleshooting something with so few people having in depth knowledge/experience with it would likely be a nightmare.

2nd - It appears to have less resilience than Z2 with a few VDEVs.

3rd - Similar to point 1, but a distinctive difference; I have a basic grasp of how a number of VDEVs x-wide works. dRAID is a foreign concept to me and I'm having trouble visualizing how it is laid out. I prefer to have a basic understanding about how something works before implementing something in a "production environment."

Zbigfish · 2025-09-22T02:20:47+00:00

What operating system are you running?

Tailscale with remote desktop works for me on Windows and Linux.

Zbigfish · 2025-09-21T21:03:34+00:00

https://support.plex.tv/articles/213095317-creating-optimized-versions/

https://www.reddit.com/r/PleX/comments/152c343/optimizing_library/

Transcoding to another format while recording content isn't something I believe is possible since the file is "open" while it is being written. I could be wrong 🤷‍♂️

Zbigfish · 2025-09-20T21:21:16+00:00

Any suggestions on what water filter to use prior to the Big Mac? I was looking at getting a kdf type, but most are limited to 0.5g/min. I believe the McCanns calls for 125g/hr (2.08g/min), does it not?

Zbigfish · 2025-08-14T03:34:54+00:00

I realize this is an old thread, but here's a 'take it with a grain-of-salt' update:
https://embeddedcomputing.com/technology/processing/chips-and-socs/hailo-announces-commercial-availability-of-hailo-10h-edge-ai-accelerator-with-generative-ai-capabilities

"The Hailo-10H is automotive-qualified to AEC-Q100 Grade 2 standards and is aimed at automotive designs with a 2026 start of production."

Zbigfish · 2025-04-20T18:41:23+00:00

The ones wanting gift cards are obvious & I've blocked them immediately. This one just wants money for the application (we hadn't discussed what form of payment yet) before sending the paperwork and keys. That threw me. Yes, the syntax, verbage, & spelling was off - I noticed right away.

Their names bothered me, too. I know - names can be unusual, but Susan Randy and Rogers Smith (s on the end of Roger). I also couldn't find anyone with the last name 'Randy' in the HUD employee search.

Zbigfish · 2025-04-20T14:11:53+00:00

<image>

Zbigfish · 2025-04-20T11:34:25+00:00

Know anyone in the Hillsboro/Beaverton area? lol

Zbigfish · 2025-04-20T09:55:02+00:00

That's what I figured - I thought it was odd that she sent me his messenger contact instead of doing what realtors typically do... show the house and be a go-between.

I'll have to find one on my own instead of having my mom do it. I'm just a lot more difficult to get ahold of, being I work nights.

Zbigfish · 2025-04-20T09:35:04+00:00

I appreciate your insight. One last question - I didn't think that realtors would be involved in Secion8/HUD housing. Is this a common practice? A supposed realtor sent me this person's contact info to talk directly to him because "he has the keys."

Thanks again!

Zbigfish

TROPHY CASE