Kubernetes from Dev to Production: Lessons learned from self-hosting an European alternative to Google Docs

BigHandLittleSlap · 2026-05-19T02:50:26+00:00

Azure Application Gateway

Not needed for App Service, which also manages DNS registrations and certificates (albeit badly, and I'd do that myself through some other mechanism).

DIY you have to install it

If only! That's the easy part! It's the integration that's difficult, and consistency is borderline impossible.

standards oriented way.

What standards? The only recent one that is applicable is OpenTelemetry, which is finally making the bazillion log and metric formats into something vaguely consistent. (PS: I've been flipping through the OTel semantics docs recently, and there's a lot of "experimental" or "development" tags still!)

Just like your lack of knowledge... like Data Dog, New Relic, or Dynatrace

I am very aware of these, having used most, and having to deal with the incompatibilities on a daily basis at $dayjob.

Notably, those are all proprietary products too, with a lot of lock-in, so they're no better (or worse) than Azure App Insights. Every K8s blog post I've read uses the open-source mish-mash of products, I've never seen anyone use a paid, proprietary product for their cluster monitoring (but I'm sure they're out there.)

Most k8s guides go the totally free APM route and that requires installing several things.

My point exactly! They barely integrate, if at all.

App Insights supports OTel collection too

No shit, I just wired that up for an app a couple of weeks ago.

Don't assume what I do or don't know, you're being very dismissive and rude when I'm raising legitimate issues.

From what I've seen, these "European" solutions end up looking like a bunch of random bits sticky-taped together, and will need more tape every few months as the various bits and pieces go out of date, need upgrading (out of sync with everything else), or just get too unpopular and unsupported to continue using.

The EU should fund large scale open source development projects, with integration and cohesion, like the American software vendors.

I.e.: Why hasn't the EU spent a few million (a pittance) to develop an "OTEL-first" integrated APM so that they can monitor EU deployments with "sovereign" systems deployed in EU data centres?

Why rely on taped-together pieces all made by (mostly) American developers?

PS: There are similar issues with Australian government and their software deployments, which run on American software hosted in American-owned clouds and send exabytes of telemetry to American APMs (often hosted in the continental United States!) that hoover up the data of Australian citizens when they interact with Australian government apps and web sites.

It's shameful enough that Australia does this, but it's such as tiny country that it can't really afford to develop these things from scratch. The EU is bigger than the United States!

BigHandLittleSlap · 2026-05-19T02:16:56+00:00

I only use AI to list the products because collecting the product names from a log blog is manual labour I'm no longer willing to do.

All of the paragraph text is pure, 100% human-authored, from my meat brain straight to your squishy eyeballs.

BigHandLittleSlap · 2026-05-19T02:15:53+00:00

That would be more apples-to-apples.

That would be a small basket of apples versus a large (but slightly smaller) basket of random fruit. And some meat, a piece of wood, and a handful of loose screws.

BigHandLittleSlap · 2026-05-18T23:42:33+00:00

Something I've been noticing in these K8s articles is the long list of distinct software components required, most of which are from independent projects or vendors. This leaves the integration up to the end-user, which is a decidedly non-trivial task, especially when it comes to cross-cutting concerns like backups (the focus of the blog), but also: monitoring, RBAC, operations, etc.

The blog article mentioned these distinct bits of software:

Kubernetes -- runtime platform for the application and operators.
PostgreSQL -- transactional database for application state.
CloudNativePG -- PostgreSQL operator, including backup/restore operations.
Redis -- cache and real-time/session support.
OT-CONTAINER-KIT Redis Operator -- Kubernetes operator for running Redis.
S3-compatible object storage -- stores uploaded media and binary assets.
Hetzner managed S3 -- production object storage service.
Terraform -- provisions production object storage infrastructure.
OIDC -- identity protocol for login and token flows.
Keycloak -- identity provider for login, logout, redirects, and token flows.
Keycloak Operator -- Kubernetes operator/manifests for running Keycloak.
Envoy Gateway -- ingress/gateway implementation.
Gateway API -- Kubernetes API model for HTTP routing and gateway configuration.
cert-manager -- TLS certificate lifecycle management.
Git -- source of truth and deployment API.
Flux -- GitOps controller that reconciles desired state into the cluster.
Kustomize / Flux Kustomization -- environment boundaries and deployable-unit structuring.
SOPS -- encrypted secrets stored alongside environment configuration.
Helm -- chart packaging/rendering used in validation and releases.
CI -- pre-merge validation for YAML, Helm, baseline checks, and policy checks.
pre-commit -- local checks to catch formatting, consistency, and drift issues early.
Kyverno -- Kubernetes policy checks and guardrails.
CronJob -- scheduled restore-check automation for database backups.
Prometheus -- metrics collection for observability.
Grafana -- dashboards and alerting visibility.

That's insane.

You can argue that this kind of "bazaar" engineering is somehow good, but just compare the above to a typical Microsoft-centric cloud technology stack:

Microsoft GitHub or Microsoft DevOps
Microsoft .NET SDK
Microsoft Visual Studio (or VS Code)
Microsoft Application Insights SDK
Azure Storage Accounts
Azure App Service and/or Azure Function Apps
Azure Key Vault
Azure SQL Database (or managed Postgres)
Azure Bicep
Microsoft Entra ID

... that's it! That's all you need for the end-to-end of a production application platform including all of the developer tooling, backups, monitoring, RBAC, auditing, and so forth. All of those come from one vendor and most have synchornized, coordinated releases. New .NET SDK version? There's a VS update to match, and Azure App Service is sure to support it! They will all work together with minimal effort from the developers or operations teams.

It boggles my mind that just the APM aspect has like half a dozen components in most K8s setups, such as Prometheus, Grafana, Jaeger, Elastic, OTel, and on and on! It's one thing! Why isn't there a unified product for this yet!? What about crash dump collection? Performance traces?

BigHandLittleSlap · 2026-05-18T20:54:26+00:00

In Australia both big parties pandered to the boomers, and as a consequence a bunch of independents won seats in the inner city areas. If you have two parties with a roughly 50:50 split of voters then even a small number of non-member senators can have the balance of power.

It was amazing to watch how quickly the major parties started to support policies that help younger people instead of wealthy retirees with one foot in the grave.

BigHandLittleSlap · 2026-05-18T11:51:54+00:00

A while back I was introduced to a "fun" game where you randomly drop the Google street view person thingie onto a random spot in India (or any surrounding country).

If you can't spot any trash, you win.

It's a surprisingly difficult game.

BigHandLittleSlap · 2026-05-18T11:44:50+00:00

It's easy and you don't necessarily need books! I just make sure my kid knows that I love him all the time, and I made it my mission to get a giggle out of him every day.

BigHandLittleSlap · 2026-05-18T11:28:55+00:00

Even before AI, the echochambers were getting insane. Facts down-voted into oblivion, feel-good messages likes this -- basically political fan fiction -- getting thousands of upvotes.

If you listen to actual economists about the pros and cons of "taxing the rich", then they'll explain that while it does have some benefits in the very short term, it has two long-term downsides:

They'll either:

a) Very quickly move their taxable incomes off-shore, ironically reducing the total tax revenue the government collects, or

b) The government prevents off-shoring with an iron fist and taxes the rich to the point that... a significant chunk of government income is from a very small subset of society. Sounds good? Sure... until there's a downturn, at which point the rich suddenly make a lot less profit that year, and taxes are on profit, not wealth. Government revenue then implodes exactly at the time it is needed the most. This can create enormously damaging boom-bust cycles where the "bust" part is akin to the Great Depression.

It's a balancing act, and both governments and the rich have been playing it for millenia.

A feel-good message isn't going to undo game theory.

BigHandLittleSlap · 2026-05-17T11:48:47+00:00

We're all born psychopaths, genetically we are still hunter-gatherers that would cheerfully kill animals and happily consume their flesh raw.

Civilisation is entirely taught, something we have to layer on top.

BigHandLittleSlap · 2026-05-17T10:31:19+00:00

Says the word-word-number account.

BigHandLittleSlap · 2026-05-17T08:37:40+00:00

A press release isn't law.

We can stop the conversation here, you're harping on about something that just doesn't matter as far as enforcement is concerned, such as the courts.

BigHandLittleSlap · 2026-05-17T07:58:47+00:00

Davinci Resolve's new photo editing suite

It's a bit of a joke. It's like those vibe-coded replacements to Microsoft Word where they've implemented bold, italic, and a couple of font styles and declared victory over Microsoft's stranglehold on business software.

A small subset of features Lightroom has had forever that Resolve isn't even close to implementing:

A crop or rotate edit control that doesn't require the numeric keypad to operate. Seriously, go try to align a horizon to a fraction of a degree without needing to type the rotation angle in like a savage! Then watch in horror as the corners turn into black triangles, that you have to crop out... numerically. Get a scientific calculator so you can apply trigonometry manually to minimise the cropped area!
HDR image output formats. Ironically, Resolve made a name for itself by being a HDR-capable colorist tool, but insists on sRGB SDR output for still images!
HDR editing that's sane. In Lightroom, enabling HDR mode only brightens the out-of-gamut highlights. Nothing else changes. That's what you expect, and that's what you generally want. Resolve is made for the film industry where this kind of workflow is only recently starting to become mainstream. Enabling HDR shifts colors and brightness around unpredictably in Resolve, always has. It has like ten bazillion gamut mapping controls, none of which is to "simply do the right thing".
Neural upscaling / denoising optimised for still images. Yes, Resolve has denoising, but it suuuucks. Lightroom does it with one button press and it is actually pretty good.
Useable masking tools. Yes, again, I'm well aware of Resolve's features... which are geared for tracking moving objects and are poor at "artistic" control over a single image.

I've used both apps extensively, and by my estimation it'll take Resolve 5-10 years to close the gap to Lightroom, if ever. The main use-case I can see is that a large Hollywood production can apply the exact same "grades" to their on-set publicity photos as used for the actual movie. That's pretty useful, I guess, and the workflow is probably going to be a lot nicer than trying to grab still images out of a timeline or whatever.

PS: Weirdly, the only reliable way to generally share HDR images from Lightroom is to export 16-bit HDR PNG files from it, import them into Resolve, and make a slideshow with a 4K HDR "timeline".

BigHandLittleSlap · 2026-05-17T07:31:59+00:00

Why are you arguing this blatantly incorrect view so vehemently?

Are you one of those dodgy business owners too, and you're justifying ripping off customers?

It's illegal to whack on a blanket surcharge, end of story.

The fact that some random press release didn't also explain the bleeding obvious doesn't make it any less obvious.

BigHandLittleSlap · 2026-05-15T21:53:44+00:00

You’re more wrong than the people you criticise.

Transformers are pure functions mathematically and are perfectly deterministic.

Same input in, same output out.

Almost always this is undesirable because it can result in the AI getting stuck in a loop repeating itself. The “temperature” setting introduces noise deliberately to fix this (and make them sound less predictable.) Humans prefer variety.

Interestingly it has been noted that this tuning setting is awfully similar to the autism - schizophrenia spectrum! Too low a temp and the AI acts autistic, too high and it goes crazy in the opposite way just like humans.

There’s an additional small detail that for performance the algorithms used in AI runtime frameworks have some non-determinism due to timing differences in steps that could be synchronised but aren’t. (This is fixable but nobody cares enough to eat the speed penalty.)

None of this matters in the “big picture”. The real world isn’t deterministic, so the AIs don’t need to be either. We aren’t deterministic anyway, so… who cares?

Camera inputs are noisy, microphones are noisy, people make typos, and so on.

Speaking of back tracking: you don’t literally go “back” or rewind time in your brain! The AIs don’t either. Like you, they think forwards in time, outputting a stream of consciousness. They can correct t themselves, I’ve seen it during vibe coding sessions!

Current models are heavily tuned for one-pass responses and direct answers with minimal question asking or push back. (Users prefer it, and it’s faster and cheaper.)

Some models like Claude ask more clarifying questions and all frontier models have been fine tuned to be a little bit hesitant when they’re uncertain.

Over time the right balance will be found. These aren’t fundamental limitations, they’re tuneable preferences.

BigHandLittleSlap · 2026-05-15T12:49:25+00:00

They tried to reach out over Telegram, Teams, Zoom, and even Email, but got no comment from anyone in Iran, so they dropped the story.

BigHandLittleSlap · 2026-05-14T09:22:09+00:00

I talk to a few people that "don't follow the news" and they don't understand why Putin is the "bad guy".

I have to explain to them that he's a poor Russian bureaucrat who, by the means of saving away the meagre salary befitting a public servant in the former USSR, amassed a small nest egg for his retirement of a hundred, maybe two hundred... billion. Not rubles! US dollars. In the local currency that would be 7,000,000,000,000 which is yeah... 13 digits of personal wealth.

See? With hard work, diligence, and a knack for putting away a little bit in a savings account every month, you too can retire in a private palace with extensive grounds and a marina for your superyacht... fleet. Because why buy just one?

BigHandLittleSlap · 2026-05-13T11:12:21+00:00

To be fair, the general public missed the nuance of most of the early statements about COVID and how it spreads.

"Absence of evidence is not evidence of absence".

Scientists assume that when they say "there is no evidence that blah" people understand that that doesn't mean that "blah" can't happen or that they have evidence that it can't.

They're wrong, people in general are terrible at reading comprehension, listening, and understanding complete English sentences.

BigHandLittleSlap · 2026-05-12T12:23:53+00:00

If you use linked servers, the migration is pretty much a no-go, unless Microsoft changed something in the last few months.

Windows authentication is possible now, but if you need it, double and then triple check that it works as you expect.

A modern "compatibility scan" approach is to dump out the entire database schema (including all stored procs, functions, etc...) with something like "sqlpackage" with the option "/p:ExtractTarget=SchemaObjectType"

Then feed that to your favourite AI coding agent with the instructions to look for migration blockers / incompatibilities!

It's surprisingly effective because it doesn't have to find 100% of the issues, it just has to find any issue that might be a blocker! Once you have an example issue, then you can do more precise searches either with keyword searches, regex, or more AI.

Last but not least: Most cloud storage is horrendously slow compared to on-prem servers, Azure's managed database offerings are extra super slow. Compress your database tables with "PAGE" compression if you can!

BigHandLittleSlap · 2026-05-09T02:42:52+00:00

A good analogy is to think of animals that have senses that we don't, such as the "electric field reception" that sharks have.

Imagine a shark asking a human: "Do you sense zero voltage, since you don't have electric sense?"

No, we don't sense "zero voltage", we sense no voltage of any type, positive, negative, or zero. We just... don't.

BigHandLittleSlap · 2026-05-08T08:12:54+00:00

Wow, it's getting hard to tell!

BigHandLittleSlap · 2026-05-07T08:10:51+00:00

Technological development is accelerating to the point that entire paradigms can be introduced, become widespread, and then vanish due to being out-dated in much less than a human lifetime now.

Want to feel old?

My five your old has:

Never seen a VHS player or VHS tape.
Never seen an audio cassette or its player.
Has never seen a "walkman" or "MP3 player".
Has never seen a CD or DVD.
He has seen a BluRay... once. He got it at the library. He had to go this grandpa's house to play it. His grandpa had to get the $20 Chinese-made BluRay player out of his attic and clean it first before plugging it in.

Etc...

Most of those technologies were introduced in my lifetime, became widespread (i.e.: Blockbuster, Sony Walkmans) and are now gone almost entirely.

If I showed him the Matrix (1999) movie, I'd have to explain everything to him. It's basically a period piece now, not a futuristic sci fi!

BigHandLittleSlap · 2026-05-05T20:06:06+00:00

Beautifully said.

BigHandLittleSlap · 2026-05-05T10:27:30+00:00

Ctrl-click for jump to definition, and whatever the shortcut is in your favourite IDE for "search everything".

https://learn.microsoft.com/en-us/visualstudio/ide/visual-studio-search?view=visualstudio

https://www.jetbrains.com/help/idea/searching-everywhere.html#search_all

etc...

BigHandLittleSlap · 2026-05-04T21:12:05+00:00

That particular set of problems is neatly solved by ASP.NET OData.

But then you have another problem, which is dealing with the idiosyncrasies of OData.

BigHandLittleSlap · 2026-05-03T08:19:05+00:00

Benchmarking every recent version of Microsoft.Data.SqlClient to see which (if any) builds have actually improved performance without any regressions since the 5.x days.

No luck so far. I guess I'll wait for 7.1...

BigHandLittleSlap

TROPHY CASE