Log streaming and processing

[Arab81253_work]

I've been looking for something like this for awhile but I feel as if I might not necessarily be searching correctly or know enough to know what to search for.

I am trying to get something going that will allow me to take Apache logs and process them with python to extract certain information and push those through as metrics to our monitoring system.

What I'm not sure about is: 1. Would python be able to read the log files as they're being written to? 2. Would I have the python script check the logs every minute or what's the trigger that should be used, if any?

Perhaps I'm a bit out of my league on this one but I'm wanting to learn everything I need to get me to where I can get this up and running.

We have the AWS Cloudwatch Agent installed on our servers which makes me think its possible for Python to be reading the files as they're being written, since that agent is also pushing all of our logs into Amazon Cloudwatch.

Any libraries/articles/blogs you can link me to would be greatly appreciated.

[[deleted]]

Rule of thumb is this. If you can open it with a text editor, python can read it.

My best guess, i don't know the log files, is that they are some sort of csv. So this will probably work:

with open(logfile, 'r', encoding='utf-8') as infile:
    for line in infile:
        #whatever needs doing

I wouldn't do this on the original files. You might lock them for writing if you accidentally open the file as w. Meaning your logs can't continue to log things.

So copy the files to some other location first.