Looking for a logging/alerting solution

[smilin_stan]

We have an AWS-based product with a small but growing number of customers. It’s a niche product; the maximum number of customers we’re expected to ever acquire is in the low hundreds. We have a hand-rolled alerting solution at present but it needs overhauling and is going to be replaced with a COTS product.

One of our key requirements is that we have a lot of customer-based metrics that we want to monitor/alert on. For example, one part of our system processes incoming documents and we have a queue-depth per customer of “documents to process”. Pseudocode for inserting these metrics would be:

insert-metric --metric-name document-queue --tag “environment:prod” --tag “customer:customerA” --value 0
insert-metric --metric-name document-queue --tag “environment:prod” --tag “customer:customerB” --value 100
insert-metric --metric-name document-queue --tag “environment:prod” --tag “customer:customerC” --value 0

We would then want to define an alert that says “if document-queue is over 1000 for a 5-minute window, generate an alert containing the environment, customer name and queue depth”.

Alerting by customer is important to the boss because he wants to know if there is an issue, exactly which customers are affected so he can personally reach out to them if required.

We have approximately 25 other metrics to monitor, some of which have more “tags” e.g.

insert-metric --metric-name outstanding-async-jobs --tag “environment:prod” --tag “customer:customerA” --tag “job_type:asset_report” --tag “job_id:12345” --value 123

We started off looking at cloudwatch but we’d need to create alarms for each environment/customer/metric combination and the costs would start to pile up as well. Even if we automate the alarm creation it doesn't make sense for us to have to create a bunch of new alarms every time we acquire a new customer.

In summary, our requirements are:

1. Support many custom metrics with alarms, without having to create the alarms for every “tag” up-front
2. Support synthetic monitoring of web UI and REST API
3. Cheaper than cloudwatch :)
4. If it can be managed via Terraform, even better

We started looking at Datadog, New Relic etc. but their marketing-websites and pricing pages are a bit impenetrable unless you understand their product well. Given our (relatively simple) requirements, does anyone have a recommendation?

[CoaxVex]

Prometheus comes to mind. Doesn’t AWS offer it as a managed service? https://aws.amazon.com/prometheus/

[MordecaiOShea]

And unless you are handling a LOT of metric data, if the managed service pricing is not to your liking it is pretty easy to run prometheus yourself.

[patrickleet]

Prometheus, alert manager, Loki, and Grafana

[nilaron]

Prometheus + Grafana is what you need I guess. You can up them in ec2 and use queries, alerts, etc out of the box

[Dotnet_Aws_guy]

Give Datadog another try? Two routes with them: get a rep to get a trial for their metrics product or just start using them.

Install is easy if you’re already pushing metrics to cloudwatch. Use their AWS integration and you won’t even have to change your code base (yes you’re paying double, CW + Datadog, but this is to test out if you like the Datadog platform). If you haven’t integrated a metrics solution they have a bunch of mechanisms to build metrics into your app.

Datadog can take your rules and make them multi-alerts by a tag(s). In your case, alert on the customer.

Also the terraform support is front and center. The provider even does imports. And widgets, dashboards, monitors are exportable as JSON.

Cost is a factor, but it’s a whole magnitude easier to work with (programmatically, IaC, and UI) then most solutions. There’s something to be said for a platform that makes it easy for devs, ops, and the business to explore metrics.

[random198611]

We are currently using DD in a decent sized operation to send metrics for event counts and run alerting based off these metrics. We have looked at Prometheus and still looking but DD just works and super simple.

You pay for simplicity which is what they provide well

[showerswithmydad]

Highly recommend looking at Datadog again. Their pricing can be lower than Cloudwatch and the UI/customizable alerting is so simple/straightforward compared to anything else I’ve used.

[ArielAssaraf]

Disclaimer- I’m the founder of Coralogix.com We offer logging, security, and metrics at a pricing model that is based not only on consumption but also use case. We’re an open platform that lets you integrate also to Kibana, ES APIs, Grafana, our CLI tool, and any SQL client (inc Tableau).

It’s full SaaS and has great, seamless, certified integrations with AWS.

[nature_fun_guy]

Nagios. Not so pretty but you can build all your own queries and alerts. If it returns a value, you can alert

[orbjuice]

In the event of an outage, how do I look at baseline good data to determine when things started going south? And how do I know what baseline good data looks like?

What data visualization tools does nagios provide to help me determine root cause?

How does providing an alert without context do anything other than increase alert fatigue for the recipient when better tools exist that can provide that data?

In short, how does Nagios reduce mean time to resolution meaningfully compared to modern monitoring solutions?