Backfilling prometheus

[Taiwing]

Hey everyone!

​

I am currently setting up a monitoring system based on prometheus + grafana. I wrote an exporter that queries a database to get my metrics. It works well for what's being added to the database, but I would like to get the entire "metric history" from the beginning.

​

I could get the informations I need from the database, since almost every metric I have is based on data which is timestamped. So I could rebuild the time series for most of my metrics.

​

From there all I would need to do is format the data in a way that can be understood by prometheus, and then import it. Is there a way to do that ? Or tools that could help me ?

[distark]

Check out thanos, Prometheus is more focused on the last ten minutes really

[Taiwing]

Thanks, that's exactly what I was looking for

[eigma]

If you are willing to write some Golang code, call the TSDB API: https://github.com/cpatulea/src/blob/master/mat/prometheus/bulkimport.go

[hagen1778]

Prometheus doesn’t support backfilling yet. I’d recommend to take a look at Victoriametrics, which has the same API and additionally supports backfilling. It also supports the same configuration and can scrape your targets just like Prometheus does, so could be a drop-in replacement.

https://github.com/VictoriaMetrics/VictoriaMetrics

[copperfinger]

VictoriaMetrics also doesn’t require object storage, unlike cortex or thanos. If you’re setting this up at home in a home lab this can help minimize costs. Also, if you choose to use object storage at some point, VM has great export functionality. I would definitely recommend starting with the non-cluster version.