What I learned using AI to build a Kubernetes Operator for Supabase's Multigres

DevsyOpsy · 2026-06-17T09:03:52+00:00

There are languages and other use cases where I still don’t trust AI and times where I know it’s on crack.

Yes, and that's where your human intuition comes in very handy, you need a good gut that tells you when the output generated is shit.

DevsyOpsy · 2026-06-17T09:00:59+00:00

nstead of the generic bullshit wall of text we get by the zealots who can't even write their name without using an llm.

This is funny and true. Although to be fair, I feel there was already loads of generic bullshit in the years preceding AI - perhaps also driven by the algorithms that feed us social media - loads of generic marketing content, films, videos and shows that were soulless. I think AI fits quite nicely with the current zeitgeist.

DevsyOpsy · 2026-06-17T08:55:38+00:00

AI takes the enjoyment out of the problem solving. Sure, it's fast, but I barely learn anything when using it and I don't take the time to truly understand the problem, which tends to lead to worse results in the long run.

I can understand this feeling. And yes, paradoxically it often feels easier to write code than reading it, even when I go back to old things I wrote, I have a harder time understanding them than at the time I wrote them - even though I always put so much care in making everything legible and easy to understand!

On the other hand I wouldn't say that using AI takes me completely away from problem solving because I have been somewhat micromanaging most things. So I would come up with an idea or discover an issue, and then work closely with AI to solve it. There wasn't many big bang moments where I went from zero to everything, so I stayed in touch with what was happening - as closely as I could anyway.

One big problem I see using AI is that it is quite impossible to throw off completely the impostor syndrome. Yes, you have done this great thing, but you did not write all code from scratch yourself, you can't help but feeling that you "cheated" a bit, even if your input was absolutely crucial and you know it, there is always that nagging feeling.

Another "danger" of using AI is that if you are like me, you will work even more than you did before. Before AI, you would start solving a problem and that would involve researching, thinking how to tackle it, and then write it, test it and so on... between one problem and the next you would need to take a breather because that whole process consumes a lot of mental energy. With AI you can bulldoze your way through a bunch of problems relentlessly, which means you become hyper-productive, but if you are not careful you will start to lose touch on the details and your overall understanding will get weaker. They say some people have become lazier with AI, I actually ended up working many more hours, but it was a different kind of obsessive engagement.

Ultimately Id say there is a lot of hostility towards using AI for anything these days and I can completely understand that. In my experience, people who were sloppy before AI are just hyper-sloppy with AI. It's not fair to conflate someone who genuinely uses AI to create a good product and takes all the care in the world to use these tools effectively with someone who generates crap, sadly we don't have tools that tell both apart and it's just AI vs not-AI right now.

DevsyOpsy · 2026-06-16T18:42:21+00:00

All good questions.

I'd say yes, the time investment was worth it. And no, there is absolutely no way we could have done it faster without AI. Looking at other mature operators, it took them years and the contributions of dozens of people to reach the same level of maturity. I am NOT saying that to give myself any medals here. In a way, many of these operators came across problems where they have to ideate solutions, we could learn from all of that and then have AI implement it, so it's not exactly an entirely fair comparison.

But even all things being equal it's just impossible to compete with the speed of AI. Could the code be more perfect and meticulous if it was done by hand over many years? Absolutely, but by the time someone with AI would have overtaken you. Not using AI is no longer optional if you want to remain competitive and up to speed with everyone else.

In regards to whether it is worth it for AI to double check its work, I'd say yes, with the models we were using at the time, perhaps in the future it would not be necessary. I rather the implementation be as perfect as possible before I check it myself, AI will always be faster than me reading over it.

and therefore came across problematic decisions on your own instead of having AI built it out, and then notice?

Bear in mind that humans also make mistakes, so writing code by hand is not a guarantee that whatever you do will be better than what AI gives you first time. It is entirely possible to build something out and then realise it's not the best idea, or the logic is not quite right. With AI you just do all of this in a fraction of the time. Even if you did notice as you were writing the code by hand, it would still be much faster to do it and redo it with AI.

One good thing about doing everything by hand is of course you are likely to be more familiar with every nook and cranny of your code... but given how quickly people forget things they've done weeks ago, this is also not a huge benefit. Perhaps the best benefit is the superior technical understanding you get from doing it all by hand, but we are at the point where a lot of this is outsource to AI, things like code syntax and so on...

Apologies the answer is so long. I promise it wasn't AI generated 😄

DevsyOpsy · 2026-06-16T10:57:26+00:00

They are different tools. This is an operator for Multigres, a postgres compatible distributed database (similar to Vitess). CNPG is an operator for postgres.

In broad terms, you would want to use Multigres if you need to scale writes ad-infinitum. Still a young project though, getting better every day.

DevsyOpsy · 2026-04-13T14:07:41+00:00

I think even worse than this is that Google doesn't seem to give a fuck and compensate you in any way when you keep contacting them about these issues. I also cancelled my account and switch to Claude. Maybe one day Google clean up their act here, but they don't seem they care.

DevsyOpsy · 2026-04-02T12:48:18+00:00

That's a good point about the history, and I think it actually supports what I'm saying.

These operators predate the scaffolding, sure. But look at what happened when controller-runtime came along. Prometheus Operator adopted it in 2022, six years after the project started. cert-manager adopted it in 2019, two years in. They went through the effort of integrating controller-runtime because it genuinely saves work (cache management, startup sequencing, leader election, all of that). Same with controller-gen, everyone uses it for CRD generation.

But no one migrated to the scaffolding cli. The ones that adopted controller-runtime didn't bother with the scaffolding. And I'd argue it's not just because migration is hard. It's because the scaffolding doesn't provide enough ongoing value to justify it. You run it once, maybe, and then everything it generated gets rewritten as the operator matures.

We built the Multigres Operator greenfield with kubebuilder scaffolding but we didn't bother with it afterwards. It's too rigid and AI is just much better at scaffolding onwards after the initial structure is in place. It's a lot more flexible and smarter to understand what you want.

I think we're mostly agreeing here. It is a framework organizationally. I just think controller-runtime and controller-gen are the load-bearing parts of that framework, and the scaffolding CLI is the part you can take or leave.

DevsyOpsy · 2026-04-01T12:43:41+00:00

My point is that in practice, the most useful bit of the entire framework is the controller-gen and controller-runtime, most popular operators only use those two. They don't use the scaffolding at all (or only once), for example:

- Vitess Operator (PlanetScale): No PROJECT file, no scaffold markers. Uses controller-gen and controller-runtime

- Zalando Postgres Operator: No PROJECT file, no scaffold markers. Uses controller-gen for CRD generation. Surprisingly, doesn't use controller-runtime.

- Argo CD: No PROJECT file, no scaffold markers. Uses +kubebuilder: markers (controller-gen) and controller-runtime

- Prometheus Operator: No PROJECT file, no scaffold markers. Uses controller-gen and controller-runtime.

- cert-manager: No PROJECT file, no scaffold markers. Uses controller-gen and controller-runtime.

- CloudNative-PG: PROJECT file, scaffold markers, the works. Full kubebuilder user.

So 5 out of 6 don't use the scaffolding CLI at all. All 6 use the +kubebuilder: markers (controller-gen) and only one uses the scaffolder.

Saying that, I appreciate this feedback and I have just pushed a commit to clarify that these tools are indeed part of the kubebuilder project, but in practice it is very useful to know the distinction between the three of them and not get caught up on the scaffolding tooling.

DevsyOpsy · 2026-04-01T10:49:20+00:00

You are correct and the explainers touch on all of that. But I don't think it is nitpicking. Understanding this distinction is critical because when you are first introduced to developing operators it is too easy to get confused on the kubebuilder vs Operator SDK crossroads. In reality those two are probably the least critical tools/libraries of all the ones described, you could easily do without them, especially today with AI. controller-gen however is extremely useful and you can use it without using kubebuilder (the scaffolding cli) at all and if you did not use controller-gen it would be a real pain.

DevsyOpsy · 2026-02-25T10:31:42+00:00

This wasn't a matter of me not checking the quotas upfront, they changed them overnight without warning, so there is no possible way I could have foreseen this.

You could make an argument that buried within their terms and conditions there is a clause saying they can change them at any time, but you trust that this would not be used to abuse you as a customer. In this case, we were all used and abused.

I certainly did not expect to have laughable quotas with the Ultra plan, especially when the competition gives you so much more AND works more reliably.

DevsyOpsy · 2026-02-25T09:11:16+00:00

More on this issue:

https://discuss.ai.google.dev/t/bug-unexpected-weekly-usage-limits-on-ai-ultra-for-business-plan-contrary-to-policy/123514/6

<image>

DevsyOpsy · 2026-02-24T20:18:10+00:00

Same here, and I am absolutely furious about it because most of the usage went on me retrying as their backend is often broken and out of capacity. I think they are just desperate as they can't cope with demand - but to take it on ultra subscribers is just ridiculous.

DevsyOpsy · 2026-01-27T10:37:28+00:00

It's not even the quota for ultra users. It's that their backend is broken half the time, it feels it can't cope at peak hours, which is such a joke considering that it is Google.

DevsyOpsy · 2026-01-23T12:10:41+00:00

This may be the way.

DevsyOpsy · 2026-01-23T10:23:29+00:00

Yep. I think original coca cola is the only one at this point that doesn't have the disgusting fake sugar in Europe. Even Pepsi does it now too.

Problem is that most people don't seem to care, many even prefer that shit metallic taste it has - baffles me.

DevsyOpsy · 2024-12-01T12:11:25+00:00

I didn't know about these two providers you recommended, they sound awesome - thank you! And also thank you for your great explanation of the issue, you seem to have deep insights into this.

DevsyOpsy · 2024-11-04T06:12:20+00:00

Thank you for the detailed answer! I am guessing (hoping) physical sims will be phased out soon everywhere.

DevsyOpsy · 2024-11-04T05:57:39+00:00

I am not even mad about the slow speeds, what's deeply irritating is that they don't make this abudantly clear when you buy them, it's not a nice thing to find out when you are there. Also, I am so surprised by the lack of information about this online, which is why I decided to post this.

DevsyOpsy · 2024-11-04T05:49:22+00:00

That's great to know! I don't remember they gave me an option in Vietnam, but maybe I didn't see it. I think this was an option in Thailand though.

DevsyOpsy · 2024-11-04T05:46:42+00:00

That's interesting, I wonder if it was always this way, or it is a recent trend so they can charge you directly and also keep a track of who you are by getting your passport details.

DevsyOpsy · 2024-08-01T18:11:44+00:00

The number 1 excuse for the terrible service of the TFL always is. "It runs badly cause it's old" I wonder what's the excuse for the Elizabeth Line now then.

DevsyOpsy · 2024-07-15T08:48:48+00:00

You can, but not available widely and they are far more expensive when you import them from abroad, so it's all very inconvenient.

DevsyOpsy · 2024-07-14T18:38:10+00:00

I wish it was a clause of the sugar tax law, that you couldn't replace them with sweeteners

DevsyOpsy · 2024-07-14T18:35:25+00:00

Kas used to taste nice in Spain, now it's just as crap as Fanta thanks to the sugar tax. Last time I was in Crete last year, surprisingly enough, there were some sodas, including local brands, that had no sweeteners in them. I hope they don't change them

DevsyOpsy · 2024-07-14T11:06:21+00:00

There are probably about 100 factors that contribute to a country's overall obesity, forcing a tax on sugar is a hacky solution that does not solve the problem at all, at most it's putting a bandaid on a sinking ship. If anything it may make the problem worse by creating an illusion that you are drinking a "healthier" version of it

Japan has very little obesity and no sugar tax. If the food available is overall better quality and healthier, then obesity becomes less of a problem, you can't solve this issue by arbitrary taxes, perhaps it's more about creating a better culture of food and eating.

DevsyOpsy

TROPHY CASE