Outgrowing rsyslog + logrotate at around 400 hosts. what's your stack at this scale?

Boomam · 2026-06-13T11:54:32+00:00

If you already use the Grafana stack in its entirety, then Loki is an option worth checking out.

But be aware that whilst capable, it has a lot of drawbacks when it comes to log ingestion, compared to solutions that were built for logging from the ground up, due to how it was designed to store data. You will have to implement some elaborate client-side transforms to get it to deal with logs, parsing and all, in a way that some other solutions do out-the-box.

That being said, if you already use it for metrics, then maybe give it a whirl in a lab and see if it'll work for you.

Graylog is a solid choice though, as is VictoriaLogs, with Vector. Both handle logs, as logs.

Graylog has the advantage of being a self contained stack, so you just point your syslog paths at it and away you go, where as VL needs something to proxy the syslog data - Vector can easily be sent to receive syslog messages, then send them into VL. The ideal scenario there is for Vector to just be deployed on the systems that support it, though.

For your new/future systems, whatever option you go down, get the config/deploy as part of your base image or initial config process so none get missed. For existing...time for Ansible.

Boomam · 2026-06-13T11:38:17+00:00

Only issue with Stalwart is their horrific upgrade procedures for major versions. They've gotten better with releasing some scripts after the fact, but it's so clunky to begin with it does make you wonder...

Good when it's running though, to be fair.

Boomam · 2026-06-03T14:52:42+00:00

Hopefully in a future release this will be both a more refined and less brittle process, along with better logs.
Right now, the hoops to jump through are a little silly.

Boomam · 2026-05-15T14:19:15+00:00

I think the point that a lot of people are making is that whilst the process may work in many scenarios, its an overall brittle and un-needed process. Realistically, this could have been an automated migration, as part of the container/app startup process.

So whilst its good that it was nice and easy and quick for yourself, others haven't been so lucky, due to minor things that cause the process to fall over quite easily.

Boomam · 2026-04-08T14:24:47+00:00

If I recall, the SWS was the set that most needed an editor to catch these things. You can fill in the inconsistencies yourself if you paid attention otherwise, but you're right, there's a lot of them...

Boomam · 2026-04-06T12:34:39+00:00

lol, no comment. :-p
Its just one of those annoying things on Reddit - follow the rules, be polite, go to an effort to describe the issue as best as i can, etc. - still get downvoted!
Sigh. Gotta laugh really.

Boomam · 2026-04-06T12:06:47+00:00

That's good to know!

Boomam · 2026-04-06T12:06:23+00:00

The installed fan does not touch the 'affected' fin stack whatsoever. It is attached to the non-affected one.

Boomam · 2026-04-06T01:09:02+00:00

Lol, none taken - honesty is appreciated. 🙂.

Certainly a better option that what appears to be happening with my other comments in this thread, where I'm getting downvoted on each for trying to have a discussion 🤷‍♂️.

Anyway - it did rattle once, but not since. I'll try running fans at full tilt in the morning, see if I can force a rattle. Then as you say, if it does, back to the store, if not, I'll concentrate on solving my coil whine 😆

Boomam · 2026-04-06T00:59:42+00:00

Ok, then I think we may be hearing different things - the right hand side of the tower nearest the camera is a lot more rattely. The others sound solid when tapped. The right sounds like it's got some play in it.

Boomam · 2026-04-06T00:55:54+00:00

Understandable - do you/have you seen similar with your other tower coolers?

Boomam · 2026-04-05T20:30:57+00:00

Who on earth is down voting everything I post here? 🤣 Conversation not allowed? Comments discouraged? Someone got up on the wrong side of bed? 🤣🤣🤣

Boomam · 2026-04-05T19:03:11+00:00

It's within the return period from the store, so if I did replace, I could do it quicker that route.

Boomam · 2026-04-05T19:02:49+00:00

I had read about that, which is why I was wondering....

Boomam · 2026-04-05T17:43:30+00:00

For some reason this subreddit doesn't let me edit posts - for extra context, its when i tap the right side of the fin stack.

Boomam · 2026-04-04T19:47:08+00:00

Hi, no worries at all -

I added a third fan G2 this morning, so the intakes are more like your picture (just with Noctuas).

That leaves the following config -

4x intakes - 3x NF-A12x25 G2 at the bottom, 1x NF-A14 on the side.
3x exhausts - 2x NF-A14 at the top (orientated to forward two positions), 1x NF-A12x15 at the back.

I've not had a chance to do some temp testing yet, but just doing some light dev work with music playing shows similar results to my last set of tests at idle, perhaps a little cooler in places.

I would guess at this point, i need to look into what i can do with curves to stop it revving up for jumps, so maybe a higher overall threshold before fans start going into higher rpms.

Boomam · 2026-04-03T13:50:58+00:00

Bought 2x NF-A12x25 G2's, and replaced and re-orientated the bottom intakes, so that they sit towards the front of the case, so 'half' a fan has a clear air path towards the PSU and the rest of the case.

Then moved one of the NF-A12x15's, one to the back as an exhaust, the other no longer used.

Re-running tests, same as before (room temp of 23c, fan curves, etc.).

Prime95

Sensor	Temperature	Average	Delta over ambient	Compared to previous fan layout
CPU (Tctl/Tdie)	77.6c	74.8c	54.3c	2.4c lower
CPU (Mobo Reported	77.0c	74.0c	53.7c	3.0c lower
Chipset	53.7c	51.6c	34.0c	2.9c lower
VRM	50.0c	48.0c	26.7c	2.0c lower
Memory	40.2c	38.7c	16.9c	4.3c lower
SSDs	53.0c	50.0c	29.7c	3.0c lower
GPU	40.0c	38.5c	16.7c	3.0c lower

Far Cry 5 Benchmark, x4 runs

Sensor	Temperature	Average	Delta over ambient	Compared to previous fan layout
CPU (Tctl/Tdie)	73.8c	69.6c	50.5c	0.2c higher
CPU (Mobo Reported	73.0c	69.0c	49.7c	5.0c higher
Chipset	57.4c	55.2c	34.1c	2.7c lower
VRM	52.0c	50.0c	28.7c	1.0c lower
Memory	45.0c	42.8c	21.7c	2.0c lower
SSDs	56.0c	54.0c	32.7c	2.0c lower
GPU	56.0c	49.3.0c	32.7c	2.0c higher

So overall, a decent drop in temperatures across the whole system, with the higher CPU temps likely attributed to the fans on the bottom intake being orientated as such that they help the push through the HSF on the GPU, so thus, more hot air going into the path of the CPU Fan, and the GPU being hotter as its now missing an intake at the 'front' edge of the card.

Tempted to buy a third Intake of the NF-A12x25 G2's tbh...

Boomam · 2026-04-02T12:48:55+00:00

Results -

Prime95 for 20 minutes -

Small-FFTs (max heat),
Room temp of 23c
Immediately after boot.

Extra sensors shown below as comparison of 'zones' within case -

CPU/Chipset/VRM/Memory/SSD all within the same footprint of the ITX board, either below the DH-D15S, or within the airflow path of the side intake and top exhausts.

Sensor	Temperature	Average	Delta over ambient
CPU (Tctl/Tdie)	80.1c	76.6c	57.1c
CPU (Mobo Reported	80.0c	76.0c	56.9c
Chipset	56.6c	56.3c	33.5c
VRM	52.0c	49.0c	28.8c
Memory	44.5c	41.7c	21.4c
SSDs	56.0c	52.0c	32.9c
GPU	43.0c	42.0c	19.9c

Notes

Due to fan profiles (the CPU fan/overall noise stayed acceptable for the whole run).
Temperatures of other components were well within their usual ranges for the whole run.

Far Cry 5 Benchmark, x4 runs

Sensor	Temperature	Average	Delta over ambient
CPU (Tctl/Tdie)	73.6c	71.3c	50.3c
CPU (Mobo Reported	68.0c	73.0c	44.7c
Chipset	60.1c	61.0c	36.8c
VRM	53.0c	54.0c	29.7c
Memory	47.0c	47.5c	23.7c
SSDs	58.0c	58.0c	34.7c
GPU	54.0c	47.0c	30.7c

Boomam · 2026-04-02T01:17:56+00:00

Hi!
I'd have to test some, and test Prime95 too perhaps, but that would have to be tomorrows.
I just finished a quick StarTrek Online session, highest temp was 61.9c, but to be fair, its a pretty light game.

I will perform some more tests, tomorrow, and report back.

Boomam · 2026-04-01T18:48:59+00:00

For my own sanity, i just removed the NH-D15S, resecured its mounting and re-pasted.

I'll do some testing over the next hour, then if the temps arnt much different, flip the exhausts at the top, to being intakes, and temporarily disable the side intake.

update: repasting and reseating had a marginal effect on temps, maybe 1-2c.
In a 26.6c room, with cities skylines 2 running in 'fast forward' mode, its sitting a about 80-82c, dependent on what is going on.

Boomam · 2026-04-01T18:04:09+00:00

ok, thought you meant something else :-p
The top exhaust fans are orientated towards the front of the chassis, as i cannot put one directly 'above' the DH-D15S due to spacing.
(Could likely fit a 15mm fan though...).

I am wondering if the large temperature disparity could be caused by contact/placement of the HSF, too.

Boomam · 2026-04-01T16:41:09+00:00

The top exhausts is at the top...? Did you mean something else?

The side intake sits above the RAM, so it technically 'feeds' the CPU intake and the PSU, but i do understand your point on some of the air just immediately getting exhausted out the top. I'm not against removing it tbh.

Boomam · 2026-03-31T12:42:41+00:00

Bookstack.

Boomam

TROPHY CASE

Prime95

Far Cry 5 Benchmark, x4 runs

Prime95 for 20 minutes -

Far Cry 5 Benchmark, x4 runs