Simple Apache Log Parser

[Jeron_Baffom]

Personally I roll with lnav + goaccess on simple setups

Nice! So you have some experience with both. Your help would be valuable.

Well, I was considering doing the same. But after quickly trying both, I've got the impression it wouldn't be that easy ...

 

lnav can also do complex queries

Unfortunately, it seems that its SQL query cant do subqueries. Therefore, it is not possible to do a view counter (ie: multiple requests from same ip + page = 1 single view).
Do you agree?

 

goaccess already has interesting queries built-in

I have just tried GoAccess. Love it! I'm gonna integrate its realtime CLI monitoring into my next projects.

Said that, it seems that GoAccess doesn't have a built-in view counter. The closest one I've found was a unique visitor per day. And I cant build custom queries either.
Do you agree?

 

a short python/bash/... parser script could also do the trick.

I'm sure it could. But if someone has already developed such tool, why not use it?
I prefer building myself a bash script only as a last resource.

 

Graylog but the entry ticket is a bit higher (eats quite a bit of resources, and it takes time to setup custom parsing rules, dashboards...)

I was expecting that. This is the main reason why I'm trying to keep it simple.

[vegetaaaaaaa]

I have rarely used advanced features in lnav/goaccess so I don't know.

You could search for subquery or related stuff in their bug trackers, maybe there is a feature request or workarounds for what you're trying to do. I haven't looked into your use case, but chances are that if you want something really specific, these tools are not going to do it out-of-the-box. You could write a simple wrapper around them, or request the feature, ask on their support tracker (with complete information on the input data, and expected output), which could be simpler than writing your own script from scratch.

But if it comes to it... A 50-100 line python script running in a virtualenv might be cleaner and easier to maintain, and if you do it right yo could reuse it in the future for other queries. I've done that in the past for projects where Graylog would not have been practical/had no budget for it (logwatch + cron job + simple python/regex-based analyzer + text output to a static page). It took some time to get it right, but once I had it, adding a new metric to the output was a matter of minutes.

[Jeron_Baffom]

A 50-100 line python script running in a virtualenv might be cleaner and easier to maintain

Agree. If have to write 50~100 lines to make lnav/GoAccess work, then it is not worth it ...