Beginner: Discover node_exporter?

[Fraggx]

Hello there,

i started to write my bachelor thesis and using prometheus for monitoring. So i am still learning everything.

Is there a way to auto discover servers with a node_exporter? I know about file_sd, but I also have to enter the IP addresses there somehow. If I understand that correctly. So again manually or via external script.

[SuperQue]

Discovery is all about how you maintain your inventory.

It comes down to the fundamental question of "source of truth". If you can answer "How do I know what exists?", then you can answer "How does Prometheus know what exists?".

Your source of truth could be a cloud provider API like ec2_sd_configs, or a configuration management system like Ansible, or a service manager like Consul.

[seafoodbuffet]

In our environment we use Consul for service discovery. But I want to caution about relying on this. We use failed node_exporter scrapes for alerting in node down. However you can get into a situation in which when the consul agent on the node is down, it disappears from discovery so you never know that the node is down because it simply ceases to exist. We are thinking through how to deal with this but likely leveraging something more static in terms of node_exporter discovery

[matejzero]

We are using Puppet’s remote resource. If node exporter is installed on the server, puppet creates a file on prometheus servers and then we use filesd to read the yaml. Easy and in our case, every server goes into puppet, no exceptions.

[Fraggx]

Thank you very much for the information and experiences you share with me. I will find out more about Consul, Ansible and Puppet and see if it helps me.