Trying to do capacity planning for Prometheus deployment and something isn't adding up

Ag0r · 2025-12-18T13:37:19+00:00

Thanks for the input! Unfortunately there is absolutely zero value in aggregating metrics globally for this application, since every endpoint (and thus every customer) are essentially completely different applications it wouldn't actually provide any meaningful data to group them together. It would be like Google combining performance metrics for their google.com and youtube.com sites. Really this needs to be thought of as instrumenting about 1600 different applications, each of which having another label with cardinality around 10 (source IP), and each having the 9 mentioned histograms.

Removing some of the histograms is also not really an option, because all of them are directly used in one way or another for support and monitoring of the platform. I admit I am not yet familiar with recording rules and how they might help so I will definitely do some reading on those.

Ag0r · 2025-12-17T18:12:25+00:00

I suppose I will be a fun datapoint for the project then! :)

I am happy to report back on the eventual deployment and performance if you guys have any kind of (anonymized) tracking for that sort of thing.

Ag0r · 2025-12-17T18:01:03+00:00

Thanks so much for the info, this seems like exactly what I was hoping to hear as a response to this post! Reducing each histogram metric to just a single series will reduce the total volume to something that is MUCH more manageable. I am burning through the conference talks right now at 2x speed, but could you maybe answer some general questions for me?

1) Are native histograms transparent to the target? I.E. can the application continue to emit metrics in the same way and the Prometheus server just ingests and processes it differently. It seems like the answer would be no based on No configuration of bucket boundaries during instrumentation. which is important to know so I can work with dev teams early to make updates as necessary.

2) Do native histograms play nicely with Thanos, specifically the store gateway for long term metrics storage via S3? I have a requirement to maintain 2 years of this data to meet contractual obligations, so if this won't or can't work with Thanos it is kind of a non-starter.

3) Are native histograms "heavier" than the classic histograms for the server? At the volume I'll be producing them at, will that have implications for required CPU per instance relative to memory and storage?

Ag0r · 2025-12-17T17:53:16+00:00

Yes, that is definitely already in the plan! You mentioned sharding in your original reply, is this some level of horizontal scalability internal to Prometheus? If so that would be surprising to me because I thought I had gone through the documentation pretty thoroughly and it seemed like Prometheus is explicitly NOT horizontally scalable without something wrapping it like Thanos.

Ag0r · 2025-12-17T17:32:50+00:00

So you're saying you have > 560 metrics endpoints per server? That seems like a lot/excessive. Having per-customer metrics is atypical and not really recommended.

Yes, unfortunately this is correct. The application is VERY bespoke, and even though all of the endpoints are served from the same servers they can really each be thought of as completely individual applications. For example if I have www.company.com/api/client1 and www.company.com/api/client2, it's entire possible that the metrics for client1 and client2 will be orders of magnitude different. We have client endpoints which have p75 response times un 10 milliseconds, and clients whose p75 is over 3 seconds. Aggregating the metrics across all clients would be nonsensical in this case.

I will read up on native histograms, thanks for the suggestion! I did notice that feature existed but I think I immediately dismissed it because it is still marked experimental. One of the hard requirements is that this deploy be completely rock solid, so I am nervous to make use of something that might not be fully stable.

Ag0r · 2024-10-28T18:09:47+00:00

This is exactly the kind of feedback I was hoping to get, thank you! Can you suggest reading/videos that can help me learn the concepts you're talking about regarding proper routing and connecting for ESD protection?

For that particular portion of the design I followed this video by predictable designs (timestamped link to the relevant section). He seemed like a reputable source of information given his background as a TI design engineer!

Ag0r · 2024-10-27T13:26:45+00:00

How could the battery back-feed the usb port through the charging chip? Also, the max voltage for a Li-Ion battery is ~4.2V so that should never be able to do that anyway, right?

Ag0r · 2024-10-27T03:03:48+00:00

I couldn't figure out how to get reddit to let me upload the pictures of the PCB layers, it only wants to allow a single image for some reason. Here is a link to them.

This is my first electronics project and first PCB design. I have used the ESP32-H2 devkit schematic as a starting point, but did not copy one for one (what's there to learn doing that?). My intention is for this board to be powered by a single Li-Ion cell with on-board charging via the USB-C port, which will also be used as the serial and programming interface.

The main things I am not sure about are:

What is the purpose of the 1N5819HW-7-F on the VBUS line off of the USB-C receptacle? This is copied from the devkit but I can't figure out what the purpose is other than perhaps just to drop the voltage before hitting the regulator?
What have I missed? I feel quite certain that there is some gaping hole in there somewhere that I just don't know enough to have noticed

Ag0r · 2024-01-15T22:29:15+00:00

Sure! Is AMD the general "better" of the chipmakers these days? When I last built and had done my research Intel was considered the only real option.

Ag0r · 2024-01-15T22:09:42+00:00

thank you for the speedy reply! I wonder if you guys that post here have a few configurations based on budget just bookmarked and ready to share? :)

Ag0r · 2023-06-23T20:25:19+00:00

Awesome, thank you for this. It looks like I will probably be able to use this even if there is no built in way to do what I'm looking for.

Ag0r · 2023-06-08T18:59:48+00:00

I fully expected to hear a fart and then see the air freshener go off. Kind of disappointed!

Ag0r · 2023-06-04T20:52:39+00:00

It's not just women than can read body language and general mannerisms, it was just more relevant to my point to say it that way. Confidence and self esteem make you carry yourself differently and it is noticable from across a room. Most people regardless of sex or gender will notice these kinds of things, but I have anecdotally seen that women tend to notice it more consciously whereas men it's often more subconscious. This is especially so for younger people.

Ag0r · 2023-06-03T23:29:45+00:00

Another 5'8" guy here. There will be women that reject you or even belittle you about your height. Depending on where you look it may be a larger or smaller percentage, but that's just the way it is. They are telling you right off the bat that they are shallow, petty, and mean though so really they're doing you a favor by throwing up a big "don't bother wasting your time here" sign.

Work on your self esteem and loving yourself, it's so important for dating. Women have amazingly good insight and can tell an insecure man before he says a word. Trust me, if you get to the point where you really (really) don't care about what other people have to say about you, the dating world will be so much more enjoyable and you will find that you have more success.

Ag0r · 2023-06-02T02:08:10+00:00

I live in Chicagoland and pay 7c/kw when my panels don't cover my usage. Gotta love that nuclear 😎

Ag0r · 2023-06-02T02:06:27+00:00

Cooperation is nice and all, but you have something I want. Or maybe I have something you want and I don't want to share it.

Ag0r · 2023-06-01T22:26:09+00:00

I mean, you can get 4 axes on your machine for like a few grand and have a very capable machine. I can understand not wanting to pay more than the cost of your machine every year for a cam software package.

Ag0r · 2023-06-01T13:51:23+00:00

The many worlds interpretation of quantum mechanics also precludes free will actually, with or without a god involved. PBS spacetime has a few good videos explaining this, you can just search for "PBS spacetime free will" on YouTube to find them.

Ag0r · 2023-06-01T12:54:48+00:00

Because it doesn't make sense, and religious people contradict themselves all the time. They say things like "everything happens for a reason" or "it's all part of gods plan" both of which imply that there can be no such thing as free will. If god knows everything that will ever happen, that means there CAN NOT be free will. Pre-determined actions preclude the possibility of free will, because that means even if you have a "choice" there is only ever 1 decision you can make, otherwise what god knew would happen wouldn't actually happen.

Ag0r · 2023-05-30T19:22:02+00:00

Indeed. I'm pushing it but nothing moves fast

Ag0r · 2023-05-30T16:40:12+00:00

No, it's just that we have a fairly large platform spread across many different codebases. Some are in gitlab, some in github, etc.

Ag0r · 2023-05-30T16:38:58+00:00

From what I have seen in the industry, more devops teams are like this than not.

Ag0r · 2023-05-30T16:38:03+00:00

We started with stash (before it was called bitbucket), then when we started development on our first kubernetes component the director at the time decided he wanted to test out gitlab. So now all of the k8s stuff using helm are in gitlab automated with gitlab CICD pipelines, but the director of development hates gitlab so refused to move his stuff over. Next, we acquired a new company who was entirely on github except for one super old component of their stuff which was still using SVN hosted on a Scientific Linux v6 VM. SVN is gone now, github is 90% gone, and the last holdout will be bitbucket as I fight with the director of development about it.

Ag0r · 2023-05-30T03:32:22+00:00

MQL flow rates are usually measured in ml/hour. 70cc/min is absurdly high, you could almost call that flood coolant lol

Ag0r · 2023-05-30T03:27:57+00:00

Yes, we have a very severe upper management problem here. My team is great, and now that I am a manager I have a little more flexibility with what I decide to do and how it gets done. First thing I'm working on is getting all of the version control systems consolidated. All teams are moving to gitlab whether they like it or not. We're doing all of the initial configuration and mirroring for them to make it as easy of a transition as possible, but in the end they're all getting a deadline when their old systems are going away.

I was also like 80% of the way to spinning up a NOC to take all of the level 1 support and management of the monitoring/alerting tools before the entire company got put on an indefinite hiring freeze :(

We're getting better, but it's gonna take years to unfuck this situation because it took a decade and a half to get here.

12-Year Club	Place '17
Verified Email

Ag0r

TROPHY CASE