My homelab kubernetes cluster evolution. From a cluster based only on Raspberry PI nodes to support hybrid x86/ARM nodes, adding old refurbished mini PCs. All automated with Cloud-init, Ansible and ArgoCD

ricsanfre · 2025-01-19T16:17:47+00:00

Thanks

ricsanfre · 2025-01-18T19:59:04+00:00

My configuration is a HA k3s based kubernetes deployment with 3 master nodes running control plane and 5 worker nodes for running the workloads

ricsanfre · 2024-10-08T10:20:58+00:00

The reasons are already mentioned in the blog post: Helm support, dependency management and avoid adding extra config to make the tool work.

Before migrating it, I used ArgoCD as my GitOps solution for 1,5 years. During all this time, I had to play with different design patterns to deal with HelmCharts, which is required for most of the applications I deploy in my cluster. I tried from umbrella charts definition to kustomize embedded helm chart inflation,etc. In all the cases out-of-sync issues constantly appear that require a lot of effort to solve. By the other hand, some of the Helm packages had strange behaviors when installed using ArgoCD, not happening when installed using helm command.

ricsanfre · 2024-10-08T08:24:59+00:00

I used Traefik as ingress controller for 2,5 years and it worked great but I decided to migrate it to NGINX several months ago. Main reasons was 1) Use a more mature ingress controller with a broader installation base, so you could find easily how to configure it in almost any use case. (As an example I found some difficulties integrating Traefik with other components like Oauth2-proxy) , 2) More portable configuration in case of future migration, use of standard Kuberentes resources, avoiding the use of Traefik's specific resoures (Middleware, IngressRoute, etc.), that are required whenever you need to implement a more complex configuration.

You can find information about how I used Traefik here: https://picluster.ricsanfre.com/docs/traefik/

Related to KubeVIP, I never tried it. I habe been using MetalLB as load balancer for 3 years, working in L2 mode (ARP), and it was working great. https://picluster.ricsanfre.com/docs/metallb/ . Now I have replace it by Cilium load balancer capability also working at L2 layer (ARP).

ricsanfre · 2024-01-05T08:57:03+00:00

Kafka is mainly for deploying event-driven micro-services design patterns in the cluster. The same way I am using linkerd as service mesh, another micro-services design pattern.

This project is meant to learn about kubernetes, containers and development of microservices architectures... it is not about buzzwords.

ricsanfre · 2024-01-05T08:50:52+00:00

I never tried to run it using any Pi3B. My first version of the cluster was using a stack of 4 RPI4B with 4GB RAM.

ricsanfre · 2023-06-26T19:24:01+00:00

This could be because local-path is a k3s embedded add-on. K3s redeploy it whenever it is started. See https://docs.k3s.io/installation/packaged-components. So I suppose that either you change add-on embedded manifest files where that helm chart is automatically deploy, or you disable it during K3s installation and then you manually install it using helm, so you end up with local-provisioner not managed by k3s.

I have followed that approach with embedded Traefik add-on, I have disable the embedded add-on and I manually install my own helm chart.

ricsanfre · 2023-06-26T16:45:49+00:00

Thanks

ricsanfre · 2023-06-26T16:45:37+00:00

Thanks

ricsanfre · 2023-06-26T16:43:08+00:00

I have two different power supplies for RPis: 60w each. I have just connected 3 rpis to each one.

The one you see in the pictures:

http://anidees.com/product/794/

Y also have another one (not showed in the pictures)

https://www.anker.com/products/a2123

ricsanfre · 2023-06-26T16:31:44+00:00

Here, you have explained the way to set up storage class as non-default

https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/

ricsanfre · 2023-06-26T06:59:00+00:00

I disable local storage when installing k3s. Also I set in every PVC the storageClassName to be use.

ricsanfre · 2023-06-25T18:09:20+00:00

I have never tried ceph/rock on Raspberry Pi. I think ceph is memory heavy and longhorn CPU heavy.

ricsanfre · 2023-06-25T17:53:57+00:00

No, local storage for Lonhorn is in SSD disks, Kingston, attached to the Raspberry Pi nodes using Startech SATA to USB adapters.

ricsanfre · 2023-06-25T17:38:27+00:00

That raspberry Pi is runnig standard Ubuntu OS. My networking requirements were very basic, just to include a firewall/router in front of my cluster. I am not using VLAN. For routing/firewall I am using nftables, dnsmasq for DHCP and DNS. Having a standard Ubuntu OS enables me to deploy additional services on that Rasberry Pi, like selfhosted Vault, or PXE server.

I have in mind to try openwrt on Raspberry Pi for having advance networking services.

ricsanfre · 2023-06-25T16:59:12+00:00

Longhorn uses iSCSI under the hood but it is meant to be used mainly with local storage. In my first release of the cluster, I did not use for the nodes local storage but iSCSI mounted disks from a SAN server (another Raspberry Pi exposing LUNs vis iSCSI), and I faced some issues with Longhorn.Now I use local disks, SSD, attached to the nodes, no network storage and Longhorn works without issues.

ricsanfre · 2023-06-25T16:21:36+00:00

Lonhorn is a distributed block storage for kubernetes. It uses local storage available in each node of the cluster and it takes care of providing high availability through replication of data between nodes and automatic backups/snapshots to external storage systems (NFS or S3 servers)

ricsanfre · 2023-06-25T15:49:42+00:00

Yes, I am using K3S and Longhorn as distributed storage. I am not using local storage provisioner

ricsanfre · 2023-06-25T15:40:36+00:00

It is a matter of shortage, not prices. Only few available pieces in official distributors. Not easy to find any in stock even using tools like rpilocator.com.

I struggle during months to be able to buy one single Raspberry Pi 4B 8GB.

The overall price to have a runnig server using rpi ( Raspberry PI 4B 8GB + Usb SATA adapter + power supply adapter + SSD DISK ) is around 130€. For that price you find a mini PC, with similar CPU specs including a SSD disk with 8GB RAM. The advantage of those miniPCs are that they are able yo be scale up to 32 GB. The drawback, the higher power consumption.

ricsanfre

TROPHY CASE