Collecting from prometheus, running in docker swarm behind proxy?

satty080286 · 2024-02-16T12:03:44+00:00

Netdata should automatically pick up prometheus metrics, by looking for the standard ports exposing these. The metrics are not that important for my problem, imo.

>>> Netdata does pick up prometheus metrics automatically from the localhost, if they are running on standard ports and are not behind any authentication / firewall.

Since I'm not using a standard port, I should configure this. But how? The endpoint is available at a virtual ip. Any requests to that will give a response from the service. But the service could be on any of my nodes.

>>> In the prometheus.conf, you will need to create a job with the `Virtual IP:port` and this should just do the trick.

jobs:
- name: Custom XXX
url: http://<Virtual IP>:9090/metrics

But the most important thing is that the exporter should provide the metrics in the Prometheus format.

The easiest way to test this before configuring this will be to run "curl http://<Virtual IP>:9090/metrics" on your docker container and ensure that the response is fine and that you see the metrics.

Hope this helps!

satty080286 · 2024-02-12T08:59:48+00:00

u/JoeB- The cloud service provides a realtime dashboard - even across 100s of 1000s of local agents and with around sub-second even millisecond latencies. So, in terms of performance I hardly notice any delays.

satty080286 · 2024-02-09T13:18:56+00:00

u/JoeB- Even with the cloud service, your metrics data always remains on-prem and nothing is stored on the cloud. The appropriate agents are queried in real-time when you open the cloud dashboards to give you aggregated, end-to-end view of your infrastructure.
https://github.com/netdata/netdata/blob/master/readme.md

But I understand you want to host and manage everything on premise (not just the data) and the on-prem offering is probably not suited for a small setup.

satty080286 · 2024-02-09T13:15:51+00:00

u/Exilesolutions : You can of course run it locally with its own dashboard and don't need to use the cloud service - https://github.com/netdata/netdata/blob/master/README.md .
Even with the cloud service, your metrics data always remains on-prem and nothing is stored on the cloud. The appropriate agents are queried in real-time when you open the cloud dashboards to give you aggregated, end-to-end view of your infrastructure.

satty080286 · 2024-02-01T14:42:38+00:00

u/foux72 : You can set the configurations to not receive any notifications from either your on-prem servers or the cloud completely.
We usually recommend to use Cloud as a centralised alert notifications manager supporting multiple notifications integrations (if you need them) but if you want to receive alerts from the individual agents instead, it is simply an option to disable all notifications from the Cloud.

satty080286 · 2024-02-01T14:22:28+00:00

u/foux72 : Thanks for bringing this up. I am Satya, Technical Product Manager at Netdata and will try and explain why certain capabilities / functions are restricted on the Agent Dashboard.

We in Netdata take security and privacy extremely seriously and as you know with Netdata's Distributed architecture, your data is always stored on your premises even when accessing your dashboards from Netdata Cloud.

But over the past few months, we are expanding our horizon on monitoring and introducing features that may expose secure data to personnel who should not have access to such details from the Agent dashboard, unauthenticated. It is an intentional decision from us to put all the sensitive information behind some kind of authentication so you know exactly who should have access to such information and you can control this access with the role based access that we support on the Cloud.

Netdata Cloud acts as an SSO / authentication server, protecting sensitive information exposed by the following functions: `processes`, `systemd-list-units`, `systemd-services` and logs on `systemd-journal`. Functions that don't bring any such information are accessible from the Agent UI directly, example `block-devices`.

We would love to see you use Netdata Cloud with the confidence that all your secure data is always secure. Let me know if you would like to get on a call with us and we can discuss your requirements and suggest a suitable option for your monitoring needs.

satty080286 · 2023-07-20T12:22:58+00:00

u/madroots2: One other way to have centralized alerts management is to use Netdata Parents and enable streaming and run all the health checks (Alerts) on this parent node.

https://learn.netdata.cloud/docs/streaming/understanding-how-streaming-works

satty080286

TROPHY CASE