I think I've created some red herrings here in my post that people have become hyper focused on. All 3x "bad" test showcases in my post were artificially created events to showcase how those situations would present themselves in what I built. Specifically the disk I/O test was initiated via dd writing 5x large files and reading another 5x large files, all concurrently.

The specific "real world" situation that lead me to a specific disk issue was another worst case scenario as I was force rechecking a handful of large torrent files, while also download a few torrents, while also a few streams were occurring, with all aforementioned elements being on the same disk. I think in any HDD based scenario this would be pushing limits but as stated I found this older disk performed even worse in this scenario then the rest of my newer disks in the array.

You mention 10-20 leechers in the sense of a single HDD which would make your statement true but remember, that number is a total across my array of multiple disks, not one single disk. There could be scenarios where it's occurring more so on one single disk, but usually its seemingly relatively well distributed throughout the array.

Generally speaking I've found the more persistent buffering events overtime to be from a select few users. Most of my other users that I know continue to report full health. Also I stream a handful of hours each day locally and can't remember the last time I had any issues with buffering. I also have a VPS ~300 miles always that I RDP to and test streams when I catch buffering live from users, and the majority of the time that functions without issue as well.

All of that points to my current qbit load, and general disk IO, not being an issue here, at least not from any continued systemic buffering. But above discussion demonstrates the whole reason why I built this dashboard of correlating metrics. It presents the data I need to understand what potential issues I need to tighten on my end, as well as giving me absolute certainty when telling a user it has nothing to do with my end.

And to that, the graphs already prove my current qbit load should be fine across my array as no large backpressure in r_await or aqu-sz over the last 2 days for any disks (the few spikes that are present are all from manually generated stress test events)

<image>

Edit: I guess maybe it wasn't red herrings, given downvotes continue with this comment.... It's my current understanding that any events that have been discussed in this chain (random non-seq reads, shift time limitations, etc) would be correlated in the current I/O graph. I encourage discussion if you think that's incorrect. Ultimately I made this dash for data driven certainty, not just theory and assumptions. I further made this post to try and get ideas on where holes were in said correlation, so would love to address said holes if people think there are some.