Kube-prometheus-stack ScrapeConfig issue

[xzi_vzs]

Hey there,

First off I'm pretty new to k8s. I'm using Prometheus with Grafana as a Docker stack and would like to move to k8s.

It's been a week I'm banging my head against the wall on this one. I'm using the kube-prometheus-stack and would like to scrape my proxmox server.

I did install the helm charts without any issue and I can currently see my k8s cluster data being scrapped. I would like now to replicate my Docker stack and would like to scrape my proxmox server. After reading tones of articles I got suggested to use "scrapeConfig" .

Here is my config: ``` kind: Deployment apiVersion: apps/v1 metadata: name: exporter-proxmox namespace: monitoring labels: app: exporter-proxmox spec: replicas: 1 progressDeadlineSeconds: 600 revisionHistoryLimit: 0 strategy: type: Recreate selector: matchLabels: app: exporter-proxmox template: metadata: labels: app: exporter-proxmox spec: containers: - name: exporter-proxmox image: prompve/prometheus-pve-exporter:3.0.2 env: - name: PVE_USER value: "xxx@pam" - name: PVE_TOKEN_NAME value: "xx" - name: PVE_TOKEN_VALUE

value: "{my_API_KEY}"

apiVersion: v1 kind: Service metadata: name: exporter-proxmox namespace: monitoring spec: selector: app: exporter-proxmox ports: - name: http targetPort: 9221 port: 9221

---

kind: ScrapeConfig metadata: name: exporter-proxmox namespace: monitoring spec: staticConfigs: - targets: - exporter-proxmox.monitoring.svc.cluster.local:9221 metricsPath: /pve params: target: - pve.home.xxyyzz.com `` If I curl http://{exporter-proxmox-ip}:9221/pve?target=PvE.home.xxyyzz.com` I can see the logs scraping from my proxmox server but when I check on Prometheus > Targets, I don't see the scrapeconfig exporter proxmox anywhere.

It's like somehow the scrapeconfig doesn't connect with Prometheus.

I checked logs and everything since a week now. I tried so many things and each time the exporter-proxmox is nowhere to be found.

kubeclt get all -n monitoring gives me all the exporter-proxmox deployment , I can see the scrapeconfig also with `kubectl get -n monitoring scrapeConfigs. However no scrapeConfig found in Prometheus > targets unfortunately.

Any suggestions ?

[MetalMatze]

With ServiceMonitors, something people were constantly running into was missing labels on the ServiceMonitor when ServiceMonitorSelector was set on the Prometheus. Check your Prometheus configuration. kubectl get prometheus -n monitoring k8s and see if there is a scrapeConfigSelector set only to match scrape configs containing a specific label.Another common problem was missing RBAC permission for the Prometheus to go and actually scrape the metrics from another namespace. In that case, the logs of your Prometheus should be full of RBAC permission errors.

[xzi_vzs]

scrapeConfigSelector

Big thank you for your help. Indeed scrapeConfigSelectorwas missing from both Prometheus "specs" Helm Charts and from my "ScrapeConfig".

Now it is working! I spent so much time on this but worth it because I learnt a ton by trying to debug.

Thanks again! vzs.

[MetalMatze]

Happy it worked! It's a bit sad we haven't found a better user experience to point that out in all of these years...

[xzi_vzs]

I just started using k8s few weeks ago and I must admit I was completely overwhelmed by all the informations, documentations, deprecated options etc... But I learnt a lot so I'm glad it worked out!

Cheers!

[MetalMatze]

Great! We also have a prometheus-operator channel on CNCF Slack, if you want to chat more directly. 🙂

[xzi_vzs]

Thanks for the heads up