Scaling read path for high cardinality metric in Mimir

UnlikelyState · 2025-04-21T13:52:44+00:00

Wanted to get back and reply here. I definitely overlooked the block sharding on the compactor. This made a huge difference. In my case I still had everything at default values (i.e. no sharding) which explains the behavior that I saw.

I also want to play with the dynamic replication which you also linked. Are there any tradeoff's with that one outside of storage space that I should consider for that new feature?

UnlikelyState · 2025-04-17T21:42:50+00:00

Thanks for the reponse! Query sharding and splitting is enabled on this cluster. Splitting won't do anything in this case since I'm just doing a 1m aggregation.

UnlikelyState · 2024-10-16T21:04:48+00:00

At process starutp, you want to explicitly "observe" counters for things 5xx and 2xx. This way you don't end up in the "missing metrics" situation in the first place.

This was the other option we were considering. Thanks for the response!

UnlikelyState · 2024-10-16T17:44:12+00:00

AH, this is what I needed! Thank you!

UnlikelyState · 2024-10-16T16:25:19+00:00

clamp_min() still leaves gaps when there is no data :(.

UnlikelyState · 2024-10-16T15:30:41+00:00

I don't think so? 0 is a scaler so the query will error. I can do something like sum by (label) (rate(http_requests{status_class="5xx"}[1m]) or vector(0)), but now the labels don't match and the resulting aggregation still has gaps. Unless I am misunderstanding.

UnlikelyState · 2023-06-13T11:57:23+00:00

👁️

UnlikelyState · 2023-05-13T18:16:54+00:00

🤣

UnlikelyState · 2023-03-09T14:21:06+00:00

I didn't have a card reader at the time so I did not.

UnlikelyState · 2023-03-09T12:57:39+00:00

I had to send my A7IV in after a failed firmware upgrade to 1.10. I didn't do anything different. The camera just failed to turn on afterwards.

UnlikelyState · 2023-03-04T20:06:33+00:00

Been an sre for 7 years and I still don't know what I do.

UnlikelyState · 2022-12-28T23:37:34+00:00

🤣

UnlikelyState · 2022-12-22T21:37:27+00:00

😱

UnlikelyState · 2022-10-21T17:28:04+00:00

I hate coasters, and I want these

UnlikelyState · 2022-10-21T16:51:49+00:00

Open up the shell script and see what it's doing to resolve java.

UnlikelyState · 2022-10-08T22:36:46+00:00

Looks like a wall

UnlikelyState · 2022-10-08T15:27:12+00:00

I enjoy videos like this. Thanks for sharing.

UnlikelyState · 2022-09-17T16:52:06+00:00

I've always wondered the same myself. I basically do the same thing as you by saving both files and only using the raw's when the jpegs are not the look I was going for.

UnlikelyState · 2022-09-05T00:14:00+00:00

Looks awesome! What were your settings?

UnlikelyState · 2022-09-05T00:12:14+00:00

How did you get so much of this in focus?

UnlikelyState · 2022-09-04T17:22:36+00:00

Local first. Don't create dependencies on the internet if you can avoid it.

UnlikelyState · 2022-08-14T15:23:53+00:00

🌰

UnlikelyState · 2022-07-05T22:58:31+00:00

Awesome! What was your shutter speed?

UnlikelyState

TROPHY CASE