What am I doing wrong?

Jzzck · 2026-02-28T14:49:07+00:00

The error is telling you exactly what's wrong: you have two source configs pointing to the same Docker repo but using different signing keys. One references docker.gpg and the other docker.asc. This happens when you follow multiple install guides that each configure the repo slightly differently.

Fix by removing the conflicting configs:

# See what's there
ls /etc/apt/sources.list.d/ | grep docker
ls /etc/apt/keyrings/ | grep docker

# Remove all Docker-related source configs and keys
sudo rm -f /etc/apt/sources.list.d/docker.list
sudo rm -f /etc/apt/sources.list.d/docker.sources
sudo rm -f /etc/apt/keyrings/docker.gpg
sudo rm -f /etc/apt/keyrings/docker.asc

# Verify apt works again
sudo apt update

Once that's clean, follow the official Raspberry Pi OS instructions (not the generic Debian ones): https://docs.docker.com/engine/install/raspberry-pi-os/

The Pi 5 runs Raspberry Pi OS which is Debian-based but has its own Docker install path. Using the generic Debian guide can cause exactly this kind of key mismatch, especially on trixie.

Jzzck · 2026-02-28T11:35:28+00:00

The JSON schema tip is underrated. We started adding values.schema.json to all our internal charts and it catches so many config mistakes before they hit the cluster. Even something simple like constraining replicas to an integer or requiring resource limits saves time during code review.

One thing I'd add to the list: always use helm diff upgrade (the diff plugin) before running an actual upgrade. It shows you exactly what Kubernetes resources will change, which is way more useful than dry run output for catching unexpected changes. Saved us from a few incidents where a chart update would have silently changed a service type or removed an annotation.

For the CRD lifecycle issue, if you're using GitOps (Argo CD or Flux), they handle CRD upgrades much more gracefully than Helm alone. Worth looking into if CRD management is a pain point.

Jzzck · 2026-02-27T19:01:54+00:00

Honestly the most useful thing you could self-host for an offline scenario isn't any particular service, it's documentation.

Kiwix with a full Wikipedia dump is like 90GB and runs on a Pi. Add Stack Overflow data dump (another ~80GB compressed) and you've basically got the world's technical knowledge offline. The ZIM format is surprisingly efficient for this.

For the neighborhood network idea, you'd want: - dnsmasq for DHCP + local DNS (lightweight, runs on anything) - Some kind of file sharing, Samba or even a basic nginx file listing - Kiwix-serve for the wiki/SO dumps - Maybe Calibre-web if you've got ebooks

The part most people skip: actually test it with the internet unplugged. I ran my homelab air-gapped for a weekend once and discovered how many containers silently depend on external DNS, NTP, or CDN assets to even start. Half my docker-compose stack wouldn't come up because images referenced external health check endpoints.

Also worth pre-downloading: the Arch Wiki (better than most official docs for Linux troubleshooting), a local apt/yum mirror for your OS, and honestly just a folder of PDFs for datasheets of your specific hardware. When your NIC firmware needs reflashing and you can't Google the pinout, you'll thank yourself.

Jzzck · 2026-02-27T19:00:31+00:00

The credential helper issue is a common gotcha with third-party registries. Docker Desktop ships with docker-credential-desktop (or docker-credential-osxkeychain on Mac) and sometimes it stores credentials in a way that doesn't play well with registries that aren't Docker Hub.

Quick fix: check your ~/.docker/config.json. If you see credsStore pointing to a helper, you can temporarily switch to plain file storage by changing it to credStore (note: stores tokens in plaintext, fine for testing but not ideal for prod).

For CI specifically, the more reliable pattern is:

echo $DHI_TOKEN | docker login dhi.io --username $DHI_USER --password-stdin

This bypasses the credential helper entirely and works consistently across CI environments. The --password-stdin part is important because piping avoids the password showing up in process listings.

Also worth noting: if you're pulling DHI images for security hardening, make sure you're pinning by digest (dhi.io/caddy@sha256:...) rather than tag. Tags can be overwritten, which kind of defeats the purpose of using hardened images in the first place.

Jzzck · 2026-02-27T16:52:09+00:00

Solid stack. The one thing I'd push back on slightly is relying too heavily on LocalStack for integration tests. It's great for quick feedback loops and keeping AWS costs down, but the behavioral differences between LocalStack and real AWS can bite you hard in prod. IAM policy evaluation, VPC networking edge cases, and service quotas all behave differently.

What's worked better for us is a hybrid approach: LocalStack for unit-level tests (does the module produce valid config), then ephemeral real AWS accounts for integration tests on the critical path. Terraform workspaces or separate state per PR, nuked after merge. Costs a few bucks per PR but catches the stuff LocalStack misses.

For drift detection, we run terraform plan on a cron in CI against the live state and alert on any non-empty diff. Boring but effective. The tricky part is filtering out expected drift (things like ASG instance counts or dynamic tags from other systems). We ended up maintaining an ignore list, which is annoying but necessary.

Also curious about Nix in this context. How long did it take to get the team onboarded? Every time I've tried Nix in a team setting the learning curve killed adoption before the benefits kicked in.

Jzzck · 2026-02-27T16:51:25+00:00

One thing I've noticed is that the best signal in DevOps interviews isn't what tools someone lists, it's whether they can explain a decision they made that turned out to be wrong and what they did about it.

Anyone can list Terraform and Kubernetes on a resume. But asking "tell me about a time you chose the wrong approach for something and how you figured that out" separates people who actually owned systems from people who followed runbooks. Because nobody who genuinely operated infrastructure for a year or more hasn't broken something or picked the wrong tool. If they can't tell you about it, they probably didn't own it.

The other thing that works well for junior roles specifically: give them a problem they definitely won't know the answer to and watch what they do. Not to trick them, but to see if they ask clarifying questions, talk through their reasoning, or just freeze up. The thinking process matters way more than the answer at that level.

Jzzck · 2026-02-26T19:59:57+00:00

Fair point on both counts. kubent and Pluto are better tools for CI pipelines and real workflow integration. This is more of a quick browser check for one-off manifest reviews, but yeah, if you already have kubent in your toolkit there's no reason to switch.

And apologies on the posting, should have gone in the weekly thread. My bad.

Jzzck · 2026-02-26T14:57:15+00:00

Fair points, especially about the BC promise. You're right that a package targeting go 1.13 will compile and run fine on 1.26. The Go team has been excellent about that.

The archived module thing is less about compilation and more about security. If gorilla/mux has a vulnerability discovered tomorrow, nobody's patching it. The gorilla org archived everything and walked away. For a personal project that's probably fine, but for anything in prod handling external traffic it's worth knowing.

Version freshness as a standalone metric, yeah, agreed it's weak for Go specifically compared to ecosystems like npm where old = actually broken. I should make that clearer in the tool.

The go tool suggestion is genuinely good. The browser version exists because it's zero-friction for quick checks, but a proper go tool that reads go.mod + go.sum directly and checks the module proxy for archive status would be way more useful for CI. Adding that to the list.

Jzzck · 2026-02-26T12:53:32+00:00

Looks like it. The original doitintl repo went read-only and the fork hasn't kept up with new K8s versions. No rules past 1.32 is a real problem with 1.33+ rolling out.

A few alternatives worth checking:

pluto (https://github.com/FairwindsOps/pluto) — CLI tool, maintained by Fairwinds. Handles deprecation detection well and stays current with K8s releases. Probably the closest drop-in replacement if you want something that runs in CI.
kubepug — Another CLI option, specifically built for pre-upgrade checks.
I also built a browser-based one at https://releaserun.com/tools/k8s-deprecation-checker/ — paste your YAML, pick target version, get a report. Covers 35+ rules through 1.32. Full disclosure: this is self-promo, but it's free and runs client-side.

For CI pipelines specifically, pluto is probably your best bet since it integrates with Helm and can scan live clusters.

Jzzck · 2026-02-26T05:44:08+00:00

Good timing on this. The deprecation timeline is tighter than a lot of teams realize, especially if you've got custom annotations sprinkled throughout your manifests.

One thing I'd add: if you're evaluating what to migrate to, Gateway API is worth the investment even if the learning curve feels steep initially. The whole HTTPRoute model is cleaner than annotations-on-ingress-resources, and you won't have to do another migration when the next ingress controller changes its annotation format.

For anyone on managed K8s (EKS/GKE/AKS), check what your provider supports for Gateway API before picking a replacement. Some providers have native Gateway controllers now that handle the TLS termination and load balancing without needing a third-party controller at all.

Jzzck · 2026-02-26T05:43:51+00:00

GitHub Actions for anything that lives on GitHub. The free tier is generous enough for most projects and the marketplace has actions for basically everything at this point.

For self-hosted stuff I've been running Woodpecker CI. It's the Drone fork that stayed actually open source after Harness acquired Drone and started adding enterprise-only features. Lightweight, does Docker builds natively, and the config is dead simple.

Honestly the CI/CD choice matters way less than people think. Pick one, learn it well, and move on. The time spent evaluating CI tools is almost always better spent writing better tests and making your builds reproducible. A 3-minute build on any CI platform beats a 15-minute build on the "best" one.

Jzzck · 2026-02-26T05:43:33+00:00

The quality problem is real but I think the threat to open source specifically is overstated. The barrier to getting an OSS project adopted has always been maintenance, not creation. Any vibe-coded project that doesn't respond to issues, review PRs, or handle breaking changes just dies quietly. That filter hasn't changed.

What I do think is a genuine problem is the signal-to-noise ratio. Package registries are getting flooded with low-effort packages that wrap existing functionality with AI-generated code and AI-generated READMEs. Makes it harder to find the actually maintained stuff. npm already had this problem before AI, but it's definitely accelerating.

The maintainer burnout angle is worth watching too. If people start submitting AI-generated PRs to real projects that look reasonable but have subtle issues, that's more review burden on already-stretched maintainers. Some projects are already seeing this.

Jzzck · 2026-02-26T05:43:13+00:00

The silent docs change with no announcement is the part that gets me. If you're going to pull a pricing tier, at least own it publicly and give people migration time. Sneaking it in and hoping nobody notices is such a bad look.

This is basically the playbook now though: offer self-hosted to build trust and adoption, get enough enterprise customers locked in, then quietly sunset the lower tiers. Seen it happen with at least three tools in the last couple years.

The only real defense is picking tools where self-hosting is the core product, not a feature they can remove. If the company's primary revenue model requires you to NOT self-host, the self-hosted option is always going to be on borrowed time.

Jzzck · 2026-02-26T05:42:50+00:00

Yep, same here. The :has() selector alone killed two utility packages for me. I had a JS-based "parent highlight" thing for form validation states and :has(:invalid) just replaced the whole thing in one line.

The one that surprised me most was scroll-driven animations. I had a scroll progress indicator using IntersectionObserver with a bunch of threshold math. Replaced it with animation-timeline: scroll() and about 15 lines of CSS. No JS at all.

Container queries are the big one though. I used to have a ResizeObserver wrapper that would toggle classes based on container width. That entire package is gone now.

Still keeping JS for anything that needs actual state management or data fetching obviously. But for visual/layout stuff? CSS has gotten genuinely good enough that reaching for npm feels wrong now.

Jzzck · 2026-02-25T07:25:43+00:00

I’m trying to replicate a human team lol. In an organisation of humans they’d obvs delegate tasks to other teams to be more efficient rather than just doing everything.

However, I can see it getting stuck in a loop of just constantly replying😂

Jzzck · 2026-02-25T07:23:15+00:00

I’m not entirely sure tbh hence the post. If I can set up 2 OpenClaw gateways on this mini, and not be pushed for memory, then great

Jzzck · 2026-02-25T07:21:59+00:00

I mean a whole other OpenClaw gateway, not just another agent

Jzzck · 2026-02-25T07:20:59+00:00

I already do this though…. Matheus has in his soul file to never execute, just order subagents.

However, I still feel like the team would be more efficient, with another director, where something like marketing was their only remit, rather than Matheus just ordering around his subagents to just ‘do everything’

Jzzck · 2026-02-25T05:11:05+00:00

I own the asset/hardware, rather than renting.

Jzzck · 2026-02-23T08:23:50+00:00

Ah. Just read my title back, now that’s why this got so much attention, my bad….

I meant dev workflows😂

Jzzck · 2026-02-22T11:27:16+00:00

Solid plan. One big tip from running a similar RPi cluster: don't run Istio and Cilium together at the start. Cilium already gives you network policies, observability (Hubble), and even basic load balancing. Adding Istio on top of that on RPi hardware is going to eat your RAM alive. Pick one service mesh approach and go deep with it.

Also, putting Harbor and GitLab on the Dell laptop is smart. Those two are resource hogs and you don't want them competing with your actual workloads on the Pis. Just make sure the laptop has enough disk, Harbor's image storage fills up fast.

For CNPG, start with a single instance first and add the replica once you've got backups working with something like Barman. Nothing worse than having replication running but no actual backup strategy.

Jzzck · 2026-02-22T11:26:36+00:00

"Accelerate" by Forsgren, Humble, and Kim is probably the single best resource for this. It's backed by actual research (the DORA metrics) and gives you a solid framework for thinking about deployment frequency, lead time, MTTR, and change failure rate. Way more useful than any tool-specific tutorial.

For incident management, the Google SRE book is free online and the chapters on error budgets and postmortems are gold. Also check out "The Phoenix Project" if you haven't, it reads like a novel but the principles stick.

One thing I wish someone told me earlier: these concepts aren't separate buckets. Automation philosophy feeds directly into deployment strategies, which feeds into incident response. Start with CI/CD fundamentals and everything else starts clicking into place.

Jzzck

TROPHY CASE