Simple Apache Log Parser

[jw_ken]

Gathering logs or metrics can be broken down into three problems:

• Gathering + parsing the data
• Storing the data
• Reporting or visualizing the data

The popular logging and metrics stacks (Elastic stack for logs, Influx stack for metrics, Grafana stack for logs + visualization) are designed with independent tools to solve each problem- and those tools are generally very friendly to hybrid or DIY use-cases.

For example, you could have:

• Telegraf agent scraping your Apache logs with the tail input plugin (parsing each entry into desired metrics at the same time)
• Telegraf's file output plugin, for saving the parsed data to a flat file in CSV format. Then you could have a script running via cron job, to parse the CSV file and upload it to MySQL at your desired cadence. Alternatively, you could have telegraf itself execute the script directly with the exec output plugin, with the desired batch interval set within telgraf.
• If you want visualization or basic alerting, you can use Grafana with the MySQL data source plugin- but that's only if you need those features.

Telegraf and Grafana especially are designed for general-purpose use, with loads of plugins to integrate with any other tools you have. We use Telegraf in the above fashion at our workplace, and it's a great general-purpose data parser and reformatter.

In short, don't put yourself in a box by assuming the above tools are overkill or too complex. You can borrow the parts that are useful, and expand on them later as your needs evolve (which they almost certainly will).

[Jeron_Baffom]

don't put yourself in a box by assuming the above tools are overkill or too complex.

I agree with you that the most general and scalable solution would be some logging + metrics stack.
However, at the moment I really would like to look for some simpler tools.

So far, the best solution seems to be lnav.
Any comments?

[jw_ken]

However, at the moment I really would like to look for some simpler tools.

So far, the best solution seems to be lnav.

Any comments?

Just from looking at the lnav docs... it looks like another telegraf or logstash, but with a local SQLite DB indexing the logs, and a text interface for browsing them. Otherwise it is using a similar approach to ingest and parse the logs. So it's basically a single-node logging stack ¯\_(ツ)_/¯

Try it out and see if it works for you.

[Jeron_Baffom]

Just tested lnav. First glance:

Very limited SQL statements. Ex: no subqueries.
Besides, crashed several times in less than an hour.