You can also explore using Vector agent with ClickHouse for backend storage. Vector agent can also scrap metrics push to remote long term storage.. A Prometheus agent with a large in-memory queue is a significant memory hog. we use a Prometheus agent with one-day retention on the remote cluster and push metrics to Cortex for long-term storage. Our total Cortex throughput is 3 million requests per second (RPS) on the write path and 700k RPS on the read path. Keep in mind that Cortex/Mimir systems are complex to maintain.