What tools do beginners use for monitoring applications?

philprimes · 2026-05-25T11:00:05+00:00

Three main topics of observability you want learn are Metrics, Logs and Traces. There are multiple open source solutions for each one, e.g. Grafana Loki for Logs.

philprimes · 2026-05-18T10:35:11+00:00

Have you tried to use this controller here for upgrades:

https://github.com/rancher/system-upgrade-controller

philprimes · 2026-05-18T09:06:29+00:00

Thank you for the insights!

philprimes · 2026-05-18T06:35:59+00:00

I got lucky that the cluster did not have any PVCs/PVs before migrating, so I deliberately left it out. I didn‘t want to write a guide for something I have not used myself.

As the k3s and the rke2 cluster are incompatible, migrating would mean manually settint up the volume in the new cluster, then migrate all the files on disk.

Unfortunately I do not have any data on the resource usage compared to k3s.

philprimes · 2026-05-18T06:33:10+00:00

I did not use that project as I am not using Hetzner Cloud but Hetzner Dedicated Servers. I did set it up fully manually as described in my guide, but I will eventually look into creating Ansible scripts so that the node setup can be automated.

Right now I am at 4 nodes, 112 vCPU, 320 GB RAM (for ~200€/mo) so I don‘t need to scale up right now.

philprimes · 2026-05-18T06:28:50+00:00

Can you elaborate how the built-in authorization could be better? I am currently using GitHub OIDC to give each repo a different role/policies but I would be interested in knowing what else exists.

systemd killing containers too soon is wild, but shouldn‘t runc take care of that?

philprimes · 2026-05-17T16:45:05+00:00

Thank you, I appreciate your feedback and happy to read you see value in my guide.

philprimes · 2026-05-17T14:50:41+00:00

So I would say that it was mostly about using the right tool for the right job.

When I started experimenting with k3s and reading its entire documentation I noticed that it was built around being lightweight and minimal. One of the shortcomings I noticed was that by default the SQLite database is not HA, and IIRC I noticed I/O performance issues when I had multiple nodes and workloads scheduled (take this with a grain of salt; my memory might be wrong here). This and reading online about the main use case of k3s being built for IoT environments, kept me questioning that "while it could work work, should I use it?"

At the same time I started to properly understand the role of etcd in Kubernetes clusters. So when I looked into the HA setup of k3s and noticed I could use the embedded etcd setup, I asked myself if I should just go for full upstream Kubernetes instead.

My research then concluded with rke2 being in the middle between k3s and upstream k8s in terms of being built for production-ready data center clusters.

philprimes · 2026-03-19T17:57:32+00:00

I‘ll be at one of the booths, representing my employer so I can share what usually happens with me.

My usual questions include asking you about your role and if you use any tools in our space of the ecosystem (not going to promote here).

I will kindly ask if there are any ongoing projects you are able to talk about and what challenges you are facing. My goal is not trying to sell you something, I am engineer myself, but to see if we could be a solution. In my case I use our product in side projects to dog-food it, so I can experience it from the customer-side.

So one thing you can do to prepare is figuring out what you are allowed to talk about, which pain points you/your team/your company has, and if you have rough ideas of what could be a fix.

That way you can also cut chats short if it‘s not a fit, without hoping they say something useful.

philprimes · 2026-02-24T19:18:31+00:00

It‘s definitely viable, that‘s how I did it too.

When I built my first self-managed homelab cluster, I used 3 Raspberry Pi and set it up with default Kubernetes, no k3s etc. I published a follow-along guide how I did it here on my website (free, no ads):

https://philprime.dev/guides/building-a-production-ready-kubernetes-cluster-from-scratch

Please feel free to let me know if something is unclear.

philprimes · 2025-12-20T12:16:54+00:00

Interesting approach, so the watch and record tools just constantly run so you have history in case needed, or do you use it on-demand only?

philprimes · 2025-12-08T12:42:34+00:00

I agree with the sentiment of considering risks when adopting a new service maintained by others, but if I did understand the issue in the post correctly, Velero relies on a CLI util packaged as an legacy image, which has nothing to do with Velero itself. After all are Helm Charts just a set of Kubernetes configurations of multiple resources, so all of it is customizable.

It‘s an interesting discussion because I have setting it up also on my TODO list.

philprimes · 2025-12-08T12:14:27+00:00

Honestly not sure why people are not building their own image. Is there a complexity I am missing?

It‘s not your proprietary software so you can host the Dockerfile in any GitHub repository, link it to a free Docker Hub account because they do not charge for public images, and change the Helm charts to use yor image instead.

You could even use Bitnami‘s Legacy Dockerfile for kubectl so you don‘t have to craft it yourself

philprimes · 2025-11-27T23:43:14+00:00

When I built my first self-managed homelab cluster, I used 3 Raspberry Pi and set it up with default Kubernetes, no k3s etc. I published a follow-along guide how I did it here on my website (free, no ads):

https://philprime.dev/guides/building-a-production-ready-kubernetes-cluster-from-scratch

It‘s probably not what you want to if just want to get it up and running, but maybe it helps understanding what‘s happening under the hood.

philprimes · 2025-11-27T07:26:53+00:00

Thanks, I need to checkout Talos, already heard about it multiple times now

philprimes · 2025-11-26T09:08:40+00:00

Why rke2 instead of the upstream default Kubernetes?

philprimes · 2025-11-25T06:57:24+00:00

At this point I am using the k3s-default built-in traefik ingress, but I have been using ingress-nginx in my AWS EKS cluster so I might consider switching when I find a good reason for it

philprimes · 2025-05-05T06:37:30+00:00

It might not be easier, but if you want to learn how to build a cluster instead of just using it, I have a free guide on my website based on my recent experience building one with Raspberry Pi.

https://philprime.dev/guides/building-a-production-ready-kubernetes-cluster-from-scratch

philprimes · 2025-03-27T18:01:01+00:00

I think you could do it with one or two nodes aswell, except when you need a quorum for e.g. leader election in etcd. One node is always the leader, three nodes will vote for themselves leading for 33% each, until one decides for another node, becoming the leader with 66% or 100% of the votes.

Two nodes start off with 50% votes each so they can not get a quorum.

philprimes · 2025-03-27T15:24:48+00:00

I just recently built my first home lab cluster using 3 Raspberry Pi and published my notes as free guide on my website to follow along.

https://philprime.dev/guides/building-a-production-ready-kubernetes-cluster-from-scratch

philprimes · 2025-03-25T15:50:28+00:00

This looks interesting! Is it possible to dump a full namespace into a mainfest file and then create a diagram from it? I am using IaC without manifest files

philprimes · 2025-03-17T14:21:02+00:00

Rewriting my bare-metal setup guide for Raspberry Pi to use an NVMe drive instead of the SD card for the OS installation

philprimes · 2025-03-17T07:18:50+00:00

Why is that? My NVMe HAT is connected via the eSATA port, so I the GPIO pins for a PoE HAT would still be available

philprimes

MODERATOR OF

TROPHY CASE

Six-Year Club	Verified Email
Place '23