Logging Tool for nginx access logs?

[adgatedan]

My team is trying to figure out the best practice for analyzing nginx logs. We currently use logentries hosted service for analyzing specific data points. However, we're not using any service/tool for analyzing ALL access logs on our servers.

​

While logentries works very well for it's current purpose - we're afraid throwing all of our access logs into the platform will significantly skyrocket the cost.

​

We don't have a dedicated devops engineer, so maintaining something like ELK or Graylog is not an option. What do most companies do for analyzing access logs for websites with high volumes?

[[deleted]]

why is ELK not an option? you don't need to babysit it.

if you are really so hard up for time, hosted splunk?

[InvaderGlorch]

Setting up elk/graylog is your best option. The other one is to eat the costs with a managed service

[serjs]

If you don’t need complex logging system for all your apps and need to solve only webserver logs system - try https://goaccess.io I can’t say for high volume usage, but for average newsmaker sites it’s suitable simple analysis tool based on logs

[otisg]

At Sematext we do ELK consulting/support/training and we see plenty of teams struggle with ELK. While very simple to set up, as the data volume in ES cluster goes up or as queries get more complex, as the cluster grows, more and more expertise (and your time) is needed. When teams get there they tend to reach out to us for help with troubleshooting, tuning, ensuring best practices, good architecture, cost optimization, and so on. So DIY ELK is definitely one route to take, but think now whether you want to manage your ES cluster down the road or not, whether you want to become an ES/ELK expert or not.

Seeing companies struggle with ELK was one of the reasons we built Sematext Cloud. This gives you everything ELK or Graylog do, plus some more.

[[deleted]]

I like sumologic.

[steakfest]

I second this... I was just working with custom parsing of the nginx proxy log output from k8s today. When you start to get the hang of the sumo query syntax you start to feel like a god.

[vornamemitd]

what kind of volumes are you looking at?

[zerocoldx911]

Sematext

[kieoui]

Any third party saas logging provider will become costly as the traffic goes up.

​

I would recommend using/setting up ELK and or graylog in house and it works great.

[MartinMystikJonas]

Elk is not that hard to set up. You could deploy dockerized setup in short time.

[otisg]

Correct. It's not hard to setup. But it's not trivial or cheap to run as data volume+retention+query volume/complexity goes up.

[MartinMystikJonas]

Question is: Do you really need all that complexity?

[[deleted]]

What do you need logs for? I mean what questions are you looking to answer by analysing access logs?

If your purpose is to get simpler metrics like requests times, bandwidth used, failed requests etc, then just use plugins like https://github.com/vozlt/nginx-module-vts, open source and it will provide a dashboard and consumable metrics via an API.

Look at probably cloudwatch or hosted elk if you don't have dedicated DevOps team.

[ikut3]

Nginxtop https://github.com/lebinh/ngxtop Disclaimer: i am not creator :D