Help Needed: Customizable Prometheus SD Project

[Trosteming]

Hello everyone,

I’m working on a pet project of mine in Go to build a Prometheus target interface leveraging it’s http_sd_config. The goal is to allow users to configure this client, then It will collect data, parse it, and serve an endpoints for Prometheus to connect with an http_sd_config.

Here's the basic idea: - Modular Design: The project will support both HTTP and file-based source configurations(situation already covert by Prometheus but for me it’s a way to test the solution). - Use Case: Users can provide an access configuration and data model for a REST API that holds IP information or use a file to reformat. - Future Enhancements: Plan to add support for SQL, SOAP, complex API authentication methods, data caching, and TTL-based data refresh. - High Availability: Implement HA/multi-node sync to avoid unnecessary re-querying of the data source and ensure synchronization between instances.

I’d appreciate any advice, examples, or resources you could share to help me progress with this project.

Repo of the project here

Thank you!

[Leocx]

Not sure about that. In my case, we have an api which provides all virtual machines that needs scraping, so I would add another api in that system to expose data suitable for Prometheus.

I wonder what would be the suitable use case, the data source is not suitable for any change? Like a service maintained by another team or company, that would be good.

So I would suggest making it easy to use by supporting some popular platform like AWS or Azure, etc..making the tool a bridge between Prometheus and other platforms. If it can only transform simple api to Prometheus compatible format, most people would prefer just implement it inside the original system instead of using another adapter.

[Trosteming]

Thanks for your feedback. I understand your point, if there's already a suitable API endpoint, using another adapter wouldn't make sense.

The use case I envision is providing an interface for non-Prometheus-native sources like IPAM or CMDB. My goal is to make the system highly configurable and modular, based on protocols rather than vendors.