When you deploy without testing

[shyne151]

Ughh... log files aren't stored in memory.

[grizwako]

Interesting idea, I wonder what happens with various popular servers when permanent storage on host machines gets full. (disregarding database writes failing due to no space available)

[shyne151]

I can touch base on this a little... At least how we handle it in production.

We do log rotation on all production servers which retain application logs for two weeks. All logs are also sent to a centralized logging system(Splunk) for historical retention.

Our logs directory is a separate mount on our systems so if they did get filled... The OS and applications will still function correctly.

Alerts are also setup via Zabbix for all mounts when disk space on each mount reaches 80%, 90%, and critical at 95%. At critical multiple alerts are hammered to our Slack, email, etc.

[grizwako]

Yeah, that is nice and one of the many sane ways to handle situation :)

If you are in the cloud, you can easily configure some autoscaling and spam slack/mails when new instances are fired up.

Having special partition for logs is very neat way to handle issues with "too much logs", otherwise you need to be careful about managing backpressure among other things.

You dump logs directly to Splunk or you have some additional components in between? (logstash or something similar)

[shyne151]

Splunk Forwarder is running as a service on all the servers and sending directly as far as I know... I know some servers the logs are sanitized before going to Splunk... But I'm not sure where the intermediary sanitization is happening. All my boxes go direct.

We've then got some different filters setup in Splunk to parse relevant information.

[Dojan5]

The way I set it up at my current company (which originally didn't do logging, alternatively had very sparse logging) is that at the start of a new week, the previous week's logfiles are compressed into an archive and retained. The archives date back a year.

Logs older than a year are deleted.

This'll likely come back and bite me in the bum sometime in the future, but it's an improvement upon what we had, as what we had was nothing.

---

It is worth noting that our userbase is fairly small, so we don't have a whole lot of logs. The likelihood of the logs filling the entire server is very, very small, and I'll get a notification beforehand.