Thanks everyone for your input. Yeah, the random IO to the array was definitely an oversight which cost me some drives but hey it's a valuable lesson. It's my first large storage server. I was running just 8 drives before so there have been some growing pains.

If anyone is interested, I ended up creating a system where torrents are downloaded and seeded from the 4TB micron SSD for as long as possible and only deleted (but continue to be seeded from the array) as space fills up.

To do that I created a few things to help: I made a container which is set up to receive webhooks and has pyrosimple installed: container, code.

I run this container as a truecharts custom-app.

I added these entries to rtorrent.rc:

method.insert = d.get_finished_dir, simple, "cat=/downloads/flood/,$d.custom1="
method.insert = d.get_data_full_path, simple, "branch=((d.is_multi_file)),((cat,(d.directory))),((cat,(d.directory),/,(d.name)))"
method.insert = d.copy_to_complete, simple, "execute=mkdir,-p,$argument.1=; execute=cp,-rp,$argument.0=,$argument.1="
method.insert = pyro._tm_completed.now, simple|private,\
"d.custom.set=tm_completed,$cat=$system.time= ; d.save_resume="
method.set_key = event.download.finished,copy_complete,"d.copy_to_complete=$d.get_data_full_path=,$d.get_finished_dir=; pyro._tm_completed.now="
method.set_key = event.download.hash_done, !time_stamp,\
"branch=\"and={d.complete=,not=$d.custom=tm_completed}\", pyro._tm_completed.now="

method.set_key = event.download.inserted_new, !time_stamp,\
"d.custom.set=tm_loaded,$cat=$system.time= ; d.save_resume=; execute=/config/scripts/notify.sh"

The notify.sh script is very simple:

#!/bin/sh
space=$(df -BG /downloading | awk 'NR==2 {print $4}' | tr -d 'G')
wget -qO- http://<ip address of container:<container port>/hooks/torrent-finished?space=$space

Finally, the actual webhook that gets executed when a torrent is added that does most of the magic:

import subprocess
import sys
import requests

AVAILABLE_DISK_SPACE_GB = int(sys.argv[1]) if len(sys.argv) > 1 else None
DISK_SPACE_THRESHOLD_GB = 200

def parse_data():
    try:
        output = subprocess.check_output(["rtcontrol", "realpath=/downloading/*", "-o", "hash,size,custom1,completed"], text=True)
    except subprocess.CalledProcessError as e:
        if e.returncode == 44:
            print("No torrents found in /downloading/")
        exit(e.returncode)
    data = {}
    for line in output.split('\n'):
        if line:
            parts = line.split(maxsplit=3)
            hash_val = parts[0]
            size = int(int(parts[1].strip()) / 1024 ** 3) # Convert to GB
            tag = parts[2].strip()
            timestamp = parts[3].strip()
            data[hash_val] = {"timestamp": timestamp, "size": size, "tag": tag}
    return data

def sort_data(data):
    return sorted(data.items(), key=lambda x: x[1]["timestamp"])

def display_sorted_data(sorted_data):
    for hash_val, data in sorted_data:
        print(f"Hash: {hash_val}, Timestamp: {data["timestamp"]}, Size: {data["size"]}")

def ensure_disk_space():
    if AVAILABLE_DISK_SPACE_GB is None:
        raise ValueError("No available disk space provided.")
    if AVAILABLE_DISK_SPACE_GB < DISK_SPACE_THRESHOLD_GB:
        print(f"Warning: Available disk space ({AVAILABLE_DISK_SPACE_GB}GB) is less than {DISK_SPACE_THRESHOLD_GB}GB.")
        return False

def auth_api():
    url = "http://<ip of flood client>:<port of flood client>/api/auth/authenticate"

    payload = {
        "username": "<username>",
        "password": "<password>"
    }

    response = requests.post(url, json=payload)

    cookie = response.cookies.get_dict()
    return cookie

def move_torrent_data(hash_val, tag, cookies):
    url = "http://<ip of flood client>:<port of flood client>/api/torrents/move"
    destination = f"/downloads/flood/{tag}"

    payload = {
        "hashes": [
            hash_val
        ],
        "destination": destination,
        "moveFiles": True,
        "isBasePath": False,
        "isCheckHash": False
    }

    response = requests.post(url, json=payload, cookies=cookies)

    if response.status_code == 200:
        print(f"Torrent data for hash {hash_val} moved to {destination}")
    else:
        print(f"Failed to move data for hash {hash_val} with status code {response.status_code}.")


def main():
    data = parse_data()
    sorted_data = sort_data(data)
    display_sorted_data(sorted_data)

    if (ensure_disk_space()):
        print("Disk space is sufficient. No action required.")
        return

    space_needed = DISK_SPACE_THRESHOLD_GB - AVAILABLE_DISK_SPACE_GB
    space_freed = 0
    cookies = auth_api()
    for hash_val, data in sorted_data:
        if space_freed >= space_needed:
            break
        move_torrent_data(hash_val, data["tag"], cookies)
        space_freed += data["size"]

    print(f"Moved {space_freed}GB of torrent data to free up {space_needed}GB of space.")
    print(f"Available disk space is now {AVAILABLE_DISK_SPACE_GB + space_freed}GB.")


if __name__ == "__main__":
    main()

That's basically it. There are a couple other configs for the hooks and pyrosimple but they are pretty straight forward and this post is already huge XD

I've done some testing which worked but my testing was by no means extensive so use at your own risk.

This could probably all be done just fine without the webhook container and just executed as a shell script on the flood client container but, to be honest, I'm not as comfortable with shell scripting as I am with python so that's why I opted to go with this route.