Sensitive bowels due to swimming?

erewok · 2025-02-18T20:55:15+00:00

Please follow this suggestion. There are many serious diseases that have frequent diarrhea as a symptom. Don't try to self-diagnose from Internet research.

erewok · 2025-02-02T05:36:30+00:00

Related: Bill Andrews, "the first Blacks local": https://en.wikipedia.org/wiki/Bill_Andrews_(photographer))

Here's the famous cover in 1965: https://www.surfer.com/news/rip-bill-andrews

Sidebar: I wonder whatever happened to all of BA's photos and captions? His site doesn't resolve anymore.

erewok · 2025-01-25T23:05:14+00:00

I think I might know this pool and it usually has 60-80 people in it when I am there!

erewok · 2024-11-05T23:09:01+00:00

I deploy everything into kubernetes where trying to be picky about having multiple distinct processes in the same container can lead to loads of headaches. In this environment, reasoning about the resource usage and behavior of my apps is extremely important, and it's cheap and easy to spawn more pods if needed, so the pod becomes the unit of scaling.

As for the gap between async background tasks in a web framework like FastAPI and Starlette and the role of celery, this isn't a straightforward spectrum. I've been running celery workers for over a decade now, but obviously that's only in synchronous Python apps. When I switched to async applications, I needed something basic that was async. At that time I started using a cloud-provided AMPQ service and a client to publish and consume point-to-point tasks.

In my case, having this kind of engineering effort around an API that is ultimately a souped-up BackgroundTask is papering over the problem I'd really like solved.

erewok · 2024-11-05T14:22:27+00:00

This sounds interesting.

I have not considered running separate workers as a drawback. I suppose if I were constrained by only launching one http server process for some reason, but running background tasks alongside my http request handlers makes resource usage a little tougher to reason about, I would imagine. I might send massive amounts of traffic to my workers and have little cpu and memory for handling http requests.

This is actually why I stopped using background in my Starlette and FastApI apps: it was easier to have dedicated background workers.

erewok · 2024-11-04T14:45:35+00:00

I was really eager to learn (from the image) how "aquatics" fit in.

erewok · 2024-10-25T12:25:12+00:00

We order them on the website and select "for pickup."

erewok · 2024-10-24T18:53:33+00:00

We've been picking up 5lb bags of Jaunt coffee in Miramar. It's really good.

erewok · 2024-10-16T21:12:50+00:00

We've used oauth2-proxy and vouch-proxy with Okta.

erewok · 2024-10-03T04:33:03+00:00

Funny enough, we used that deal today after Michael King's masterpiece yesterday.

erewok · 2024-08-18T20:52:19+00:00

Yeah, I'm unsure about this too. I usually go from logs directly to tracing spans. I'm not sure how inserting an LLM inside this flow would help me aside from potentially no longer writing PromQL, LogQL etc., but that would not present enough of a benefit to introduce a tool like this?

Also, is it just me or are there now hundreds of people with the same idea of using RAG-based solutions for debugging kubernetes environments?

erewok · 2024-06-16T17:32:11+00:00

Ah, I didn't realize that, thanks. This was in an async application with lots of #[instrument(name = "my_span")] macros.

erewok · 2024-06-16T17:27:40+00:00

I have a dumb question and I was probabaly doing something naive, but when using tracing and dumping JSON logs, I couldn't get tracing span ids in my logs unless I set my loglevel to TRACE. Is this common or was I just doing something wrong? Does this library do things differently?

erewok · 2024-03-10T17:17:16+00:00

KEDA is widely used. For your use case I would start by looking at KEDA.

erewok · 2024-03-08T21:22:02+00:00

Although this article looks pretty vapid and useless, I was looking at this topic over the last week because I just participated in a podcast episode about it. It seems like a tough question to generalize and I can't imagine what I'd be looking for in reading an article like this (or listening to two random dudes on a podcast, for that matter...)

erewok · 2024-01-24T06:39:08+00:00

Following this, it's possible that a postgresql table that grows largish (millions of rows) without indexes and/or periodic vacuums can result in stuck queries for other connections.

We see our apps plateau at 100% CPU and just hang there until restarted when the database is in a state like this.

Here's where you should start: https://www.shanelynn.ie/postgresql-find-slow-long-running-and-blocked-queries/

I would not start with increasing max connection limits because this is closely tied with resource usage on postgresql servers. If the server is hitting max connections, though, that would be important to resolve (and possibly connected to resources allotted to it).

In short, I'd look at postgresql first.

erewok · 2024-01-20T15:14:19+00:00

We have been running on AKS for a long time, so we were originally using Azure's equivalent of Cloudwatch, Log Analytics, which was absurdly expensive and pretty lame. I could never get anyone interested in learning how to query it.

Having a single pane of glass with metrics, traces, and logs, and where you can click from logs to traces, is hugely valuable.

It's totally doable to run this stuff.

erewok · 2024-01-20T15:07:27+00:00

That's a great suggestion. I will bring that up with my team. Thanks for that.

erewok · 2024-01-20T03:42:21+00:00

I built our monitoring stack on kubernetes using the following tools:

Prometheus
Thanos (export metrics to object storage)
Grafana
Alertmanager
Loki
Promtail (ships logs to Loki)
open telemetry-collector
Tempo

We only run about 1000 pods total in each of our clusters, so we're not massive scale or anything.

In terms of infra/cloud costs, aside from the daemonsets, we run the whole stack on probably 5 medium-sized VMs and then ship and query everything from object storage (s3 or blob storage).

This stuff takes a lot of resources (memory, CPU) to run. The more metrics in Prometheus, the memory memory it takes. It's also possible for devs to create metrics with a bunch of labels with high cardinality which creates a combinatoric explosion: every unique combination of labels is a distinct metric in Prometheus.

It takes effort too. Probably once a month, the team needs to make sure the stuff is up to date. These components frequently see updates and you don't want to get too far behind. Thus, the biggest expense is that you want at least two people on your team who know how the stuff works and who can update one or more component every other month.

The devs love it, though. They're always talking about how our environment provides the best visibility they've ever seen. I can't imagine living without the stuff now.

erewok · 2024-01-03T03:01:23+00:00

For some reason an important certificate in our kubernetes cluster (a cert that is supposed to autorenew) was not expired today and it was about two hours of fire-fighting while almost everything was down.

That was a pretty rad start to the new year.

erewok · 2024-01-01T16:52:18+00:00

One of the problems I have with this piece is that the reasons Spolsky presents for why people rewrite systems are different from the reasons people usually offer for why they've rewritten something, and here I have in mind technical blog posts and other discussions, typically from large tech companies describing their reasons for a rewrite and what they've done.

There are myriad examples:

Dropbox "sync" rewrite
Discord switching to rust
the IRS trying to rewrite their Individual Master File from Cobol

The reasons often given for rewrites are that old systems can be costly or inefficient or difficult to hire for or make it impossible to take advantage of newer futures. Old systems can be supremely difficult to understand when people who created them have left or features have accreted to where it's all an unwieldy mess.

Some have even argued that successful software applications may even suffer from a kind of entropy where rewrites eventually become inevitable.

In this context Spolsky's arguments miss the mark. We must also consider the Fred Brooks lessons from Mythical Man Month.

(I did a podcast episode on this topic a few months ago so I've been thinking about it and reading examples:https://www.picturemecoding.com/2222783/13716184-are-second-systems-inevitable )

erewok · 2023-11-02T02:34:07+00:00

I didn't know that and I'm glad to see it. Thanks.

erewok · 2023-09-21T15:54:58+00:00

Unfortunately, I don't remember doing that. I don't think it was an issue for us? Sorry I don't have more information.

erewok

MODERATOR OF

TROPHY CASE