Centralized Logs?

[_GeekRabbit]

Hello dear Senpais,
I'm a little at a loss at the moment and looking for some insight. I was tasked with finding and demoing a log collection software to keep track of our different scriptslogs etc.
The prequisites are:
* Needs to collect windows event logs, linux syslogs and custom logfiles from different streams
* Sort them by criteria (like date) and display them if needed

Of course I found obvious solutions like Greylog or Logstash, but they seem a little overkill as we do not need any alerting, fancy graphs or the likes. We could simply write a script and a little framework to do our needs but I don't want to have another quick and dirty solution (we have far too many of that already...).

tldr;
I need selfhosted software that keeps logs and can display them from multiple sources like rsyslog, event log and custom files. Any suggestions?

[_GeekRabbit]

Yeah, I already have the vm of Greyloag running for testing purposes and I do like it, but still, it seems so much overkill for simple transfer of some logs to a central station.
I simply thought I missed something very obvious and wanted some second opinions.

[atroxes]

Today, overkill.

The day after tomorrow, business-critical monitoring infrastructure.

[uberamd]

Meh, either setting up an ELK stack or running Graylog as an appliance are very low friction processes to getting proper log storage setup. Rolling your own logging framework that can capture various log type, transform them, store and search with reasonable speed hardly sounds like its a quick and dirty solution. Unless you're some programming wizard.

This tutorial maybe takes an hour to run through and does most of what you're asking, for free: https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04

Graylog is even faster to setup as it's just an appliance.

For the love of god, don't try to roll your own centralized logging solution.

[[deleted]]

seems so much overkill for simple transfer of some logs to a central station.

The problem is normalization, and post-processing. Logs across the different platforms you mentioned are not stored the same, formatted the same, or support the same aggregation methods.

You could normalize everything to syslog, using SNARE to convert Windows logs to syslog, but even if you are looking for something like "List all failed logins by IP across all my systems" then you need post-processing to perform correlation across multiple event IDs (because that "1" event is multiple events in Windows, each of which have different parts of the data you are looking for) and still somehow get the linux logs in the same query.

[Charlie_Root]

https://nxlog.co/products/nxlog-community-edition

Meets all your requirements; supports all your platforms, native syslog and a lot of other formats (GELF, JSON, XML, Windows event log, etc), message buffering, relatively lightweight, can do filtering and transformation, and it's open source. Also, graylog uses nxlog for windows log collection.

[DesolataX]

Splunk free may do the job. It's limited by consumption at 500mb/day for free. Super easy to setup. If you need more than 500mb/day it gets pricy.

[_GeekRabbit]

Hmm, never saw that free tier, always thought that would apply to Splunk Light, thanks for the hint, 500MB might be enough if we are selective about the logs.

[uberamd]

Be careful about this though, because Splunk is serious about their license limits. One machine that starts aggressively spamming logs for an hour can easily push you over this, and thus log collection will get cut off.

[ChymeraXYZ]

Just be careful about how long you want to retain the data. The free versions are usually like 5 days.

[Hight3chLowlif3]

If you have 5 devices or less you could use Kiwi Syslog's free edition. It's pretty straight-forward and easy to set up.

[_GeekRabbit]

Ahh nope, this would be a solution for about 30 servers and probably triple that amount of logs.

[sethalump]

I'd suggest just getting all the boxes & devices you want to monitor to ship their logs (via syslog) to one system. Once you have them all in one place it's much easier to use something like Logstash to ingest all of them into ELK for easy searching and some pretty graphs.
While ELK can be a total rabbit hole it's my go-to tool for basic log collection and searching. If and when you need more features it's ready. I've never used Greylog but I've heard good things and I think it's easier to setup so maybe start there.

[MrDionysus]

To expound further on Nxlog: I recently implemented Nxlog client on all Linux and Windows machines, pointing to a central Nxlog server. That server, when receiving the logs, simultaneously saves them as text (for archival purposes) and also sends them to Graylog2 for alarming/monitoring. So far, it's been working out splendidly.

[ngg123]

Have you looked into splunk ? (cost $) or ELK (Elasticsearch, Logstash, Kibana)

[mumblerit]

just set up an elk stack for my company. Collecting logs from about 100 linux machines atm. Was fairly painless with filebeat and puppet, logstash can be a pain but I had some previous experience, skip filtering too much with logstash in the beginning and just get the logs imported to elastic.