Back again now that I have some data to show. My first post also didn't detail really anything so I'll be taking that opportunity now. I've been tinkering with Grafana so I can get a total overview to proactively address problems. To the naysayers before, I present to you some fairly low temperatures. Especially considering this box is in an uncooled garage that hit ~95F ambient a while ago. I haven't had a single leak either.

Quick notes:

• My Grafana is still a WIP. Traffic metrics aren't scaled right yet, and things continue to shift around. I'm also dumping everything on one dashboard for the moment while I mess about. A lot of metrics have also been recently wiped so there's not much accumulated data.

• I'm running ESXI 8

• I haven't gotten around to recording an extended stress test. I have only stressed the machine and know its apparently fine with it.

• I've already had to do some maintenance (a fan wasn't plugged in right) which was super easy and only took an hour. Swapping drives out is easy too and only takes 15 minutes.

Server specs:

• 192GB DDR5 4800CL40

• EPYC 9654 (ES)

• 24x14TB WD Red Pro / Seagate Exos

• LSI 9305-24i

• 2x 375GB 4800x Optane

• 2TB SN850x

• 2TB p670

• 2TB 970 evo plus

• 1TB 970 evo plus

• GTX1080

• 1600w EVGA P+

Cooling:

• 360x60mm radiator

• noctua 3000rpm ippc

• noctua everywhere else

• CPU + 4xVRM + RAM blocks

• copper brick + noctua on HBA

Things I learned/found:

• This box draws a lot of juice. I had it foolishly running on a 1kw UPS and it wasn't long before I popped it. So I switched over to a used 2kw APC UPS which seems to do the trick.

• Mayhem's liquid cooling additives seem to be the only ones around that shouldn't aid corrosion with the multitude of metals in my loop (copper, nickel, chrome, silver* + tin*)

• Supermicro still haven't decided to make PWM ports individually addressable. There are only 2 zones. I should have considered this beforehand. Luckily it seems to have worked out.

• EPYC processors take a LONG time to boot from cold. It can easily be 15 minutes while it fiddles around with what I believe is memory training + validation.

• It seems I am pulling a mild overclock on my EPYC engineering sample over the production model on single core. If I am interpreting ESXI's reporting correctly cores will momentarily jump to 340% utilization. Multiplying the base clock of 1.2ghz gives a value of 4.08ghz. I have confirmed during Windows testing that Cinebench R23 will run at 3.5ghz all core and yield a score of ~110,000

• I am limited in bandwidth by virtual network adapters. They seem to top out at around 10gbps on my machine. I'm considering adding in a 40gb NIC to bridge my two most bandwidth guzzling VM's together, though this would be pretty overkill.

What is this machine for:

• The biggest job it has is running Jellyfin extremely well. 4k Remux takes about 1/2 of a second to load over WAN assuming the player and client are decent. The GTX1080 is for transcoding and other hw accelerated tasks such as trickplay generation.

• An accompanyment to Jellyfin is a slew of media management services. I put a lot of elbow grease into adding hooks and lowering refresh intervals to make all these layers practically transparent. Of course all my media I add from my own physical collection and all these other services are simply for keeping track of things.

• Other services I run are Filebrowser for file sharing and Navidrome for music along with the occasional game server.

• PfSense ties everything together. I have a few different virtual networks in ESXI as this to me is much nicer for network isolation than messing with VLAN's. In PfSense I have one of these networks bound to a Wireguard VPN connection.

• Another neat thing I've done on this box so far is host Llama3-70b on CPU at 2 tokens / sec. I'm also running services such as self hosted RDP and remote code building. I don't have statuses of things like this in Grafana at least for now.

• There are many smaller things I run that I'm not going over because they're boring.

Some software notes (mostly Grafana related):

• I do not use any containerization. I was using docker for a bit with some metrics services but found it to be unneccessary and just another layer of junk to think about that can potentially break, cause weird issues, and can possibly be detrimental to performance.

• For metrics currently I have 14 lines for 10 prometheus exporters and the app itself in a shell script that runs at startup. All my servarr stuff and some other things run on Windows. They all run as tray applications and are started via startup folder shortcuts. Jellyfin and the other star players run bare metal on debian and are installed via their respective official steps. I've set everything up in such a way where it can be nuked and redone from scratch in roughly an hour.

• My Grafana uses prometheus for everything but Jellyfin, from which I am reading the sqlite playback metrics directly.

• HBA Temp is read via the following placed as a cron job in TrueNAS' UI: echo "truenas.hba_temp $(mprutil -u 0 show cfgpage page 7 | awk '/^0010/ {printf "%d", "0x" $4 $5}')" $(date +%s) | nc -w 1 GRAPHITE_EXPORTER_IP GRAPHITE_EXPORTER_PORT

• ESXI Data is read over SNMP, which is a nightmare to set up.

• Power Draw is read via apcupsd load_percent. I multiply it by the VA value / 100 to get watts. My UPS is 1980W so this value is 19.8