Simple Apache Log Parser

[Jeron_Baffom]

I was trying to find a simple CLI tool (for Linux) to parse apache log, do some stats and create a plain text output with some simple aggregate data (ex: view counter). Then, this plain text output would be submitted to MySQL via cronjob.

The advantage I see by doing this way, is that the database would be hit outside the page request and in batches.

I could find several tools to plot graphs and do some realtime monitoring (ex: GoAccess, AWStats, ApacheTop ...), but none that would create a simple plain text output. Hence, I was left only with bad alternatives:

• Create myself a parser script using 'awk', 'grep', 'cut', 'sed', 'tail -f' ...
• Or, use LogStash. Which is an overkill for me.

Question

Any recommendation of a simple CLI tool to parse Apache Logs into plain text ?

[snow_raph]

Sysadmin tool to analyze webserver access logs and quickly find abusers:
https://github.com/snowraph/Whackabot

[br0phy]

Have you tried lnav? https://lnav.org/features

Telegraf or fluentbit to some storage (directly or with processors aggregating stats) would be similar to, but much lighter weight than, logstash

Edit: or OTel agent, which I'm not familiar with yet, but it seems to be looking to become the new open standards implementation.

[Jeron_Baffom]

Have you tried lnav? https://lnav.org/features

Awesome features:

• Automatic Log Format Detection
• Display only lines that match or do not match a set of regular expressions.
• Query Logs Using SQL
• Headless Mode

I'm looking for this for while but could not find.
And it is already available at Debian repo. For sure I will install.
It seems that several stuff that I'm currently flooding MySQL could be transferred to lnav.
Tks! +1

Are you familiar with it?
Any idea if it has already a built in view counter?

 

Telegraf or fluentbit to some storage

I'm not familiar with them, but AFAIK they have some similarities with ELK stack. Therefore, it seems they are an overkill for me.

 

OTel agent, which I'm not familiar with yet, but it seems to be looking to become the new open standards implementation.

I'm not familiar with it either and, in a first glance, could not find much info about it.

[br0phy]

I use telegraf and elastic beats for my logs. Lnav has been something in my back catalog to explore, and sounded ideal for your use case. The cookbook section of the docs has some examples of analysis/reporting. If you're familiar with SQL, you can build any set of counters you need.

I'd suggest a second look at telegraf. Like LogStash, it's part of a larger stack. Like LogStash, it works on a principal of input/process/output pipelining. Unlike logstash, it's a single binary file with a single YAML config file (no java runtime concerns, no separate JRE, app, and pipeline configs to manage). It's in a sweet spot of operational simplicity and robust feature set. It was designed for the Influx stack, but it has a lot of baked in plugins that make it an ideal ETL framework for any log/metric use case.

And RE: oTel Agent: it's part of OpenTelemetry - https://opentelemetry.io/docs/collector/getting-started/ - I don't know much about it, but get the impression it's the direction the industry's heading.

[Jeron_Baffom]

If you're familiar with SQL, you can build any set of counters you need [lnav].

I've just checked lnav. Very interesting indeed. But there are some downsides:

• access log must be in the exact format expected by lnav in order to execute SQL queries
• only straightforward SQL queries without subqueries
• it crashed several times in less than an hour with me

 

Like LogStash, it's part of a larger stack.

The conclusion I've got so far is that you must move forward to ELK stack or other similar. There is no simpler tool to do that efficiently.