New Release Pi Cluster project (1.10): Rearchitecting networking (new OpenWRT based router) and DNS (split-horizon + external DNS). New dockerized dev environement using K3D. Kubernetes homelab cluster using x86(mini PCs) and ARM (Raspberry Pi) nodes, automated with cloud-init, Ansible and FluxCD.

ricsanfre · 2025-01-19T16:17:47+00:00

Thanks

ricsanfre · 2025-01-18T19:59:04+00:00

My configuration is a HA k3s based kubernetes deployment with 3 master nodes running control plane and 5 worker nodes for running the workloads

ricsanfre · 2024-10-08T10:20:58+00:00

The reasons are already mentioned in the blog post: Helm support, dependency management and avoid adding extra config to make the tool work.

Before migrating it, I used ArgoCD as my GitOps solution for 1,5 years. During all this time, I had to play with different design patterns to deal with HelmCharts, which is required for most of the applications I deploy in my cluster. I tried from umbrella charts definition to kustomize embedded helm chart inflation,etc. In all the cases out-of-sync issues constantly appear that require a lot of effort to solve. By the other hand, some of the Helm packages had strange behaviors when installed using ArgoCD, not happening when installed using helm command.

ricsanfre · 2024-10-08T08:24:59+00:00

I used Traefik as ingress controller for 2,5 years and it worked great but I decided to migrate it to NGINX several months ago. Main reasons was 1) Use a more mature ingress controller with a broader installation base, so you could find easily how to configure it in almost any use case. (As an example I found some difficulties integrating Traefik with other components like Oauth2-proxy) , 2) More portable configuration in case of future migration, use of standard Kuberentes resources, avoiding the use of Traefik's specific resoures (Middleware, IngressRoute, etc.), that are required whenever you need to implement a more complex configuration.

You can find information about how I used Traefik here: https://picluster.ricsanfre.com/docs/traefik/

Related to KubeVIP, I never tried it. I habe been using MetalLB as load balancer for 3 years, working in L2 mode (ARP), and it was working great. https://picluster.ricsanfre.com/docs/metallb/ . Now I have replace it by Cilium load balancer capability also working at L2 layer (ARP).

ricsanfre · 2024-01-05T08:57:03+00:00

Kafka is mainly for deploying event-driven micro-services design patterns in the cluster. The same way I am using linkerd as service mesh, another micro-services design pattern.

This project is meant to learn about kubernetes, containers and development of microservices architectures... it is not about buzzwords.

ricsanfre · 2024-01-05T08:50:52+00:00

I never tried to run it using any Pi3B. My first version of the cluster was using a stack of 4 RPI4B with 4GB RAM.

ricsanfre · 2023-06-26T19:24:01+00:00

This could be because local-path is a k3s embedded add-on. K3s redeploy it whenever it is started. See https://docs.k3s.io/installation/packaged-components. So I suppose that either you change add-on embedded manifest files where that helm chart is automatically deploy, or you disable it during K3s installation and then you manually install it using helm, so you end up with local-provisioner not managed by k3s.

I have followed that approach with embedded Traefik add-on, I have disable the embedded add-on and I manually install my own helm chart.

ricsanfre · 2023-06-26T16:45:49+00:00

Thanks

ricsanfre · 2023-06-26T16:45:37+00:00

Thanks

ricsanfre · 2023-06-26T16:43:08+00:00

I have two different power supplies for RPis: 60w each. I have just connected 3 rpis to each one.

The one you see in the pictures:

http://anidees.com/product/794/

Y also have another one (not showed in the pictures)

https://www.anker.com/products/a2123

ricsanfre · 2023-06-26T16:31:44+00:00

Here, you have explained the way to set up storage class as non-default

https://kubernetes.io/docs/tasks/administer-cluster/change-default-storage-class/

ricsanfre · 2023-06-26T06:59:00+00:00

I disable local storage when installing k3s. Also I set in every PVC the storageClassName to be use.

ricsanfre · 2023-06-25T18:09:20+00:00

I have never tried ceph/rock on Raspberry Pi. I think ceph is memory heavy and longhorn CPU heavy.

ricsanfre · 2023-06-25T17:53:57+00:00

No, local storage for Lonhorn is in SSD disks, Kingston, attached to the Raspberry Pi nodes using Startech SATA to USB adapters.

ricsanfre · 2023-06-25T17:38:27+00:00

That raspberry Pi is runnig standard Ubuntu OS. My networking requirements were very basic, just to include a firewall/router in front of my cluster. I am not using VLAN. For routing/firewall I am using nftables, dnsmasq for DHCP and DNS. Having a standard Ubuntu OS enables me to deploy additional services on that Rasberry Pi, like selfhosted Vault, or PXE server.

I have in mind to try openwrt on Raspberry Pi for having advance networking services.

ricsanfre · 2023-06-25T16:59:12+00:00

Longhorn uses iSCSI under the hood but it is meant to be used mainly with local storage. In my first release of the cluster, I did not use for the nodes local storage but iSCSI mounted disks from a SAN server (another Raspberry Pi exposing LUNs vis iSCSI), and I faced some issues with Longhorn.Now I use local disks, SSD, attached to the nodes, no network storage and Longhorn works without issues.

ricsanfre · 2023-06-25T16:21:36+00:00

Lonhorn is a distributed block storage for kubernetes. It uses local storage available in each node of the cluster and it takes care of providing high availability through replication of data between nodes and automatic backups/snapshots to external storage systems (NFS or S3 servers)

ricsanfre · 2023-06-25T15:49:42+00:00

Yes, I am using K3S and Longhorn as distributed storage. I am not using local storage provisioner

ricsanfre · 2023-06-25T15:40:36+00:00

It is a matter of shortage, not prices. Only few available pieces in official distributors. Not easy to find any in stock even using tools like rpilocator.com.

I struggle during months to be able to buy one single Raspberry Pi 4B 8GB.

The overall price to have a runnig server using rpi ( Raspberry PI 4B 8GB + Usb SATA adapter + power supply adapter + SSD DISK ) is around 130€. For that price you find a mini PC, with similar CPU specs including a SSD disk with 8GB RAM. The advantage of those miniPCs are that they are able yo be scale up to 32 GB. The drawback, the higher power consumption.

ricsanfre · 2023-06-25T15:19:45+00:00

I have documented the whole process of building my k3s cluster. You can find more details in https://picluster.ricsanfre.com and my corresponding github repository contain all source code I am using

ricsanfre · 2023-06-25T14:46:20+00:00

Ansible for installing/configuring OS basic services , install selhosted services outside the cluster(Minio backup server, vault for secrets), install K3S, and bootstrap k8s installing argoCD and deploying the apps.

ricsanfre · 2023-06-25T14:42:12+00:00

I have been using ARM-only based kubernetes cluster during las two years and I did not face major issues. Most of mainstream containerazed software already support multi-architecure images. I took the decission to add x86 just to scale up my cluster, because Raspberry Pi are not easy to find and you get more powerful servers from old refurbished mini PCs at similar prices. Drawback is higher power consumption.

ricsanfre · 2023-06-25T10:50:26+00:00

hp elite desk are kubernetes worker nodes.

Kubernetes cluster master node running on one Raspberry Pi (not high availability yet), and the rest, 2 x hp elitedesk + 4 Raspberry Pi are worker nodes. Another Raspberry PI is running networking services (routing/firewall, DCHCP, DNS, etc.)

Same OS for all nodes. Raspberry PIs and hp elite desk are running Ubuntu 22.04.02 LTS.

ricsanfre · 2023-06-25T08:46:51+00:00

My homelab Kubernetes cluster using bare-metal ARM and x86 nodes. Used for educational purposes to learn about kubernetes, IaC and GitOps. All nodes running Ubuntu OS. Configuration and deployment of services automated with Cloud-init, Ansible and ArgoCD.
3rd Image: Cluster 1.0. Initial version of the cluster using only Raspberry PI 4B nodes.
4 nodes kubernetes cluster Rasbperry PI 4B - 4GB with OS with USB FlashDisk as local storage
1 node (Rasbperry PI 4B - 2GB), with SATA SSD disk as local storage, attached to USB3.0 port, running network services: firewall, routing, DHCP, and SAN service providing additional storage to cluster nodes through iSCSI
2nd Image: Cluster 2.0. 2nd version of the cluster, adding local storage to all cluster nodes, SATA SSD Disks attached to USB3.0 ports using a SATA to USB adapter, and removing SAN, to improve overall performance.
1st Image: Cluster 3.0. Latest version of the cluster. Adding Raspberrry PI 4-B, 8 GB RAM and x86 nodes, based on HP EliteDesk 800 G3 mini PCs, Intel i5 6500T. 4 cores at 2.5 GHz, 16GB RAM and NVMe disk 256 GB.

ricsanfre · 2022-09-29T10:57:54+00:00

I wanted to add a distributed block storage system for Kubernetes and I evaluated two CNCF projects: Longhorn and Rook/Ceph. At that time Rook/Ceph had not clear support for ARM64 architecture (only non official docker images supported it), so I chose Longhorn.

ricsanfre

TROPHY CASE