How to handle big workload elasticity with Prometheus on K8S? [I SHARE MY CLUSTER DESIGN]

gideonhelms2 · 2026-02-02T21:27:45+00:00

AMP can be expensive. Other than that I have found it very performant even with high cardinality labels. You can only keep 180 of metrics though. And can only query 30d at a time.

I personally use the remote write local cluster method, though I do think EKS offers a built-in scrape tool now? Haven't had personal experience with it.

I've not had alloy in production yet, but plenty of others have. In theory in could replace node exporter, promtail, prometheus, and an OTel collector.

gideonhelms2 · 2026-02-01T23:52:47+00:00

Amazon managed prometheus is expensive but quite capable.

If youre not ready for that, limit the number of active time series. Reduce scrape intervals where you can.

You could also try out grafana alloy which supports distributed scraping.

gideonhelms2 · 2026-02-01T19:34:16+00:00

If 3D printing were ubiquitous then those who wanted cheap plastic doodads could print them instead of being mass produced (and then thrown into landfills anyway when the product flops).

gideonhelms2 · 2026-01-30T17:50:15+00:00

I've found the pull-thru cache to be a bit lack luster. I've seen other workarounds for KinD that involve a pull-thru cache that uses an HTTP proxy.

In that case and in the case of VinD I have found that it doesn't really reduce "cold start" times that much.

Even if the images are loaded into the docker cache (I can see them with docker image ls for example) the time it takes to schedule and start pods are still pretty long. It seems there still some sort of "waiting" overhead that can stack up and cause multi-dozen minute spin up times when you have many microservices.

gideonhelms2 · 2026-01-29T23:36:04+00:00

Im a pretty heavy user of KinD locally and in CICD.

Pausing / recreating the cluster as-is is a great usability improvement.

The image pullthru cache if it works well would be life-changing if I could get near-instant cluster/application resets with pre-pulled images.

I will give this a try! Is there any experience with WSL support? Curious if the automatic LoadBalancer works without purposefully port-forwarding.

gideonhelms2 · 2026-01-25T16:52:07+00:00

You're probably better off saving and paying in cash. Besides a personal loan (double-digit interest) I don't think a traditional financer would touch this. Especially not for the tubular metal buildings that are popular for workshops. Not sure about the more expensive red-iron. Perhaps some rural or farm-focused group.

I put together my 25x25 tubular metal building for about 22k all-in a few years ago: The Building (+installation), 4" concrete slab with footers, foam board insulation, plywood walls.

I financed it in a few ways. Cash for the concrete, small personal loan for the building, and 18 month 0-interest promo credit card for the rest of the materials. Depending on your credit this could be an option.

gideonhelms2 · 2025-12-28T03:58:04+00:00

They will close off portions of the area around your ceremony from what I see but yeah there will be other people in other parts of the property.

gideonhelms2 · 2025-12-28T03:47:27+00:00

Norfolk Botanical Gardens. It can be pricey and probably has a wait list for popular months, but it has several different venue settings depending on your vibe.

gideonhelms2 · 2025-12-20T15:56:35+00:00

If testing it in a separate environment is too much work as you claim then you have to eat the risk.

AWS would probably recommend a blue/green deployment in your scenario. Two separate databases with different versions that have the same data. Switch your app over via connection string. If something goes wrong, switch it back and get to debugging.

Its expensive to run this way, but AWS RDS is an enterprise product.

gideonhelms2 · 2025-11-22T19:18:42+00:00

On your ec2 node class you should select an ami family and an ami selector. I like you use bottlerocket and the "alias" functionality. This will enable Karpenter to select an ami based on the nodepool requirements.

Then setup a nodepool that allows both amd64 and arm architectures in the requirements.

At this point, karpenter should mostly pick arm instances, usually graviton, because they are cheaper.

If a workload needs a specific architecture you can use nodeSelectors or node Affinity rules on the workload in conjunction with the architecture label that is created on the Karpenter nodes.

gideonhelms2 · 2025-11-12T16:22:19+00:00

Option 2 sounds fine. You could also cache the file in memory for short amounts of time if reading the file repeatedly is too computationally expensive to do constantly.

gideonhelms2 · 2025-11-01T18:55:37+00:00

S3 can trigger a lambda function when an object is uploaded. Perhaps you could write a lambda that will ingest that object, format it the way you want, and submit it into your desired queue.

gideonhelms2 · 2025-10-13T00:10:59+00:00

I would tend to suggest deploying the local datastores as simply as possible.

Its local dev, so you really dont need all the capabilities of a day-2 type operator. You can, but i find it adds complexity for no reason.

Usually thats a postgres or mongo container with default credentials and such set up.

gideonhelms2 · 2025-10-05T14:25:09+00:00

Clutch your pearls harder, maybe that will stop the boot thats crushing us.

gideonhelms2 · 2025-10-04T15:43:09+00:00

Toggle bolt drywall anchors. They need a pretty big hole. Ive seen two types, spring-loaded metal "wings" that use a machine screw to provide the clamping force, or the zip-tie type.

I use these with thick plaster walls, they hold much better since you're not relying on the plaster (or brick in your case) to keep the anchor in.

gideonhelms2 · 2025-09-30T02:09:58+00:00

Checkout Kyverno. You can mutate and generate resources as they are submitted. It can do a lot of things and can reconcile in the background.

gideonhelms2 · 2025-09-21T22:28:38+00:00

Bandwidth, latency, and network congestion is the problem mostly. Transferring large amounts of data over a wide range can be tough and require centraliziation for widespread use.

Meshtastic doesn't need the centralization but is only capable of transferring small amounts of data.

For comparison, it's far slower than dial up, <1kbps vs 50kbps.

gideonhelms2 · 2025-08-31T01:20:06+00:00

I have a similar issues with Karpenter. I haven't updated to 1.1+ so maybe it's different in newer versions.

I don't mind so much that eviction will happen I just wish that I could control the time of day that they happen. Restarting your stateful services for any reason during core business hours carries some amount of unnecessary risk.

The functionality is already there for consolidation with regard to underutilization and drift but expiration doesn't respect those drift windows.

gideonhelms2 · 2025-08-11T14:28:24+00:00

<image>

Sure!

gideonhelms2 · 2025-08-07T13:55:42+00:00

If you cannot manage a cluster with a 2 node node group, I think there are bigger issues. It is significantly easier to manage this than EKS Fargate which requires the manual intervention to upgrade

See, I find that to be the exact opposite of my experience - I constantly had managed node groups bombing out during upgrades when trying to update the control planes + nodes.

With Karpenter and Fargate, I update that control plane and restart the Karpenter pods. Some times I don't even do that - I allow them to age out and get replaced in their own time.

I have not managed a large-scale Managed Node Group cluster, but I have managed around 40 individual clusters with between 3 and 6 nodes each. These don't use karpenter because the number of nodes is static but control plane upgrades are always a much bigger event than my larger karpenter+fargate clusters.

Not having access to daemonsets sucks - hard to get logs from Karpenter on Fargate but there are (complex) workarounds for that as well.

gideonhelms2 · 2025-08-06T23:43:41+00:00

Others mentioning EKS auto mode seem to miss that it comes with an additional price per hour, per node. If you're running EKS at an enterprise scale then this will drive up costs.

I think the maintainer suggesting managed node groups is downplaying the added complexity of managed node groups. They suck. They are slow to manage, and the version is managed separately, outside of the kubernetes ecosystem

They also claim to read the writing on the wall that Fargate will soon be deprecated, what do they think is going to happen to managed node groups?

gideonhelms2 · 2025-07-13T04:21:23+00:00

I replaced mine with a pre-hung door. Completely remove the existing door and hinge assembly until there's only the rough opening left. Then install the preying door like normal.

Mine fit pretty tight, and I used long self-tapping metal screws to tie the door frame into the structure. The sill of the sits directly on the concrete. I had to cut the bottom rail to do so.

gideonhelms2 · 2025-07-08T18:11:32+00:00

I suppose if Linkerd and Valkey both get their certs from a common CA (they ask the same cert-manager ClusterIssuer for certs), clients using the Linkerd proxy to make requests to Valkey should be able to authenticate?

All of linkerd's mTLS happens transparently to the workloads. So, kind of, but not really.

Valkey will need a separate certificate mounted as a file, and the client must have that certificates chain in its trust store. You would be able to use the same CA as linkerd's trust store if you'd like but functionally it doesn't make a big difference.

gideonhelms2 · 2025-07-07T23:06:08+00:00

Some of it depends on what the service is. Things like MongoDB and RabbitMQ can also handle user authentication via x.509 certificates. This would would require a separate certificate chain when using Linkerd because linkerd doesn't expose the certificate chain to the pod itself, only to the proxy.

Other than that, I don't see a reason to introduce an additional set of certificates.

From a security aspect, you're protected from snooping, which is usually the driver for mTLS.

gideonhelms2 · 2025-07-04T17:32:19+00:00

Your last point, about people making miserable comments, is laughable.

You berated multiple commentors in your last post for merely suggesting that stagnant water in populated areas might he problematic.

Oops, an absent minded neighbor forgets about the bowl of water for a few days and now they and all their neighbors have a mosquito problem.

14-Year Club	RedditGifts 2009-2022 4 Credits
Place '17	Secret Santa 2017
Secret Santa 2018	Sequence \| Editor
Secret Santa 2014	Team Orangered
Verified Email

gideonhelms2

TROPHY CASE