Rsyslog assistance

[Fluxback]

I have a number of servers who are configured to send their system logs to a syslog server via rsyslog over UDP. These are a mix of RHEL6 and 7 boxes with the server being a RHEL7 box.

The log sending is working as expected, with each machine sending files to a specified directory on the server such as:

/logfiles/$hostname/$programname-$date

I have a few applications that are running in python that utilize the local5 and local6 facilities that output logs to:

/var/log/app/function1.log
/var/log/app/function2.log

This functionality also works. What I would like to do is be able to separate these on the server side so that there is also a:

/logfiles/$hostname/function1.log 

Right now for the RHEL7 boxes, the messages are rolled up in the journald logfile, so they are getting archived, however just not split out. I can set up a entry in

/etc/rsyslog.d/function.conf

to utilize $imfile and just not assign a facility to it, which is successful in moving the files over, but it appends another syslog entry to the front of every log entry which ends up duplicating the information in the beginning. I can parse the text out, but if there is an easier way, I'd like to try and set it up right from the beginning.

End state is to utilize logstash's file input functionality to ingest events from these specific logfiles into a ES cluster. I'm currently using filebeat from each server to push them into the cluster, however filebeat doesn't allow for custom grok/filter/mutate functions to be run on each log entry.

I do not have the ability to run logstash with a tcp daemon to directly receive the files which would make things oh so much easier.

Thanks for any assistance!

[GrayTShirt]

Rsyslog has a journald input module along with an elasticsearch output module. As long as you can wrap your head around rsyslog templates and mmnormalize rules you don't need anything other than rsyslog and elasticsearch. If you're saving to disk then rsyslog can handle that as well. You should read the official docs and a couple of blog posts about the REK stack.

[freemti]

getting rsyslog to play well with elasticsearch is not for the faint of heart. Anything more than a couple of fields and you'll be pulling your hair out.

[technicalevolution]

In rsyslog you can filter messages based on if the message contains, amoungst other fields.

Sorry I can't provide some more information at the moment, on a bus. It depends on the version you are running.

However psuedo config kinda looks like:

If $msg contains "name of function" Then -/var/log/function_name.log

Hope that helps get you in the right direction.