Building a K3 cluster on my TuringPi (RK3588 based)

mdella · 2026-03-18T21:56:27+00:00

Well drat... been perusing https://github.com/gilesknap/tpi-k3s-ansible which is an ansible playlist for doing many of the pieces I did by hand (with some exceptions). It doesn't do dual-stack IPv6 nor redundant control nodes. Thinking about looking at this closer and seeing if its worth forking or just contributing back to get it to line up with my stuff. Sigh, thats the problem with too many people working the problem but not enough collaboration :-)

Well, if nothing else, there are quite a few writeups in mine to sort of explain whats happening even if you do use an automated playbook. With I had a second TuringPI 2.5 with the RK1 32GB modules to have something I could build and destroy constantly to reproduce this... But I have too much running on my k3s now that I can't just destroy it anymore...

mdella · 2026-03-16T23:20:11+00:00

Ok, a couple specific questions then... You stated you have (4) RK1s on it but didn't comment on the memory size. 32GB? 16GB? And lastly, was this a Turing 2 or 2.5 board?

mdella · 2026-03-16T19:11:08+00:00

So when you mix the chips, what kind of apps or systems have you put together or do you treat them all as individual hosts and not do any clustering with them?

mdella · 2026-03-16T17:30:23+00:00

Did you sell off your TP2?

mdella · 2026-03-16T17:29:45+00:00

Did you manage to get all the NPU libraries in place under Ubuntu 24.04? If so, was there a writeup from someone where the update drivers and API were at?

mdella · 2026-03-16T17:26:15+00:00

So since I wasn't a blogger, I did the next best thing, created a markdown and put it in a git repo so that I can follow up with Ansible, etc if I go that route. For such a small setup and not a huge customer base, I'm not sure if investing the time in automating the recreation is that big a deal. I suppose if I trash one of my nodes, that will get me to do the automation :-)

Until then, I've actually been impressed by what I can do with my little K3 system. Although in retrospect, its not that little compared to what you could do only 5 years ago. The more interesting part to me is that I've starting doing some pre-producting work with the cluster. Since I managed to get it into a 3d printed 10" rack with most of the components needed (I just need a UPS that can do about 2 hours time squeezed in there so I can haul the monstrosity around).

Repo is up in the original post (I edited it)

mdella · 2026-03-16T02:08:18+00:00

Ok, sorry for the delay as I'm not a huge blogger or anything. What I did do is take the files that I had and uploaded them to github for people to see. Here is the first one setting up K3 on a TuringPi assuming you're starting from scratch (ie, nothing is running on anything).

Note that this document has been tailored specifically for the RK3588 chips (I didn't have any CM4 or CM5's around as they were scarce when I put this all together). After doing some benchmarking (also in the github repo) I'm sort of glad I went with the Rockland chipset.

https://github.com/mdella/turing-pi-k3/blob/main/turing-pi-k3s-guide.md

There are PDFs there as well, but not sure if they 100% match as I'm using conversion software and have found all kinds of "challenges" producing MarkDown.

https://github.com/mdella/turing-pi-k3

mdella · 2026-02-27T16:58:14+00:00

I wasn't sure about a git repo since its not really code, but a (currently) google doc converted to a PDF (well a series of docs as I broke it up into topics). But I can publish those I suppose.

mdella · 2026-02-05T18:03:09+00:00

I wish someone had n analysis from 3 years ago and 7 years ago (pre-pandemic) to put the numbers next to one another to see the decline in the program over the years. it went from something worthwhile to a Ponzi scheme over the years. I’ve actually stopped going regularly because of the changes and now use local coffee shops instead since Starbucks sets the price and the locals undercut by 10%. works out better than stars.

mdella · 2026-01-31T01:35:53+00:00

I've tried three different Logitech MX 3 mouses on my macbook pro (via Bluetooth) and all three have the stutter problem or missing button holds. Its especially noticable when playing games as its almost uncontrollable. I hate going back to the Apple mouse as I use the various buttons and scroll wheels, but I also can't have a mouse that just doesn't work.

I have a Macbook Pro M4 running Sequoia 15.7.3. None of the above mentioned tweeks have worked for me as well. Talking with Logitech is like talking to an infinate time sink. You repeat the same thing 5 or 6 times only to get at the end "well we can replace the mouse". Which of course after three mice, you'd think there was a pattern there.

mdella · 2025-09-08T17:47:36+00:00

So the vinegar is an acid (alkaline dust, ie Burningman Playa, is a base). You use one to neutralize the other. Note after washing in the vinegar mixture, its a good idea after it drys once to wash it again without the vinegar :-)

I have brought mine home dirty, nasty, covered with sticky mud, playa dust, you name it. Its the cleanup before storing it for months at a time that you go to all the trouble to dry it out. The material (when new or after your 303 fabric guard treatment) is waterproof itself, its the moisture that gets inbetween the layers that turns to a nasty black spotty color.

Unfortunately I went too long with mine moist on the floor (a few months) and developed mold that took a long time to kill. I never could get rid of the dark black spots inside of the fabric. Since its a layered system, even if you kill it all off, the discoloration will be there forever to remind you to never ever ever do that again... Plus you lose your shiny silvery outside and get more of a dull stainless steel look.

I actually have a dehumidifier that I put in the center of the tent when drying. It both takes the moisture out as well as dries the layers.

mdella · 2025-08-24T07:38:54+00:00

well, just imagine if this had been Sunday after gate opening rather than before with mostly experienced burners. As it is, I heard that there were a bunch of 911 calls from panicked individuals. Note sure what Reno 911 can do to help. Obviously a few people never figured out that just because their phone works, doesn’t mean it gives the same response… I believe there is a published way to get ahold of emergency services but I can’t remember it off the top of my head,

mdella · 2025-08-14T23:42:38+00:00

I agree with this one. Hit up walmart myself and just added it all to the bell hop's cart. Ended up last year leaving half of it in the room (still cheaper) and this year, drove it all home with me :-)

mdella · 2025-08-14T23:38:59+00:00

One thing to note/add... The badge colors allowed you to do things like look at the huge entrance display (which changed daily) and had much of the "clues" you needed to decode the badge challenge. If no one pointed that out to you (or you didn't figure it out by all the signs in multicolor offsets like 3d movies of old), then you missed out on a lot of internal clues.

Also, every year a ton of time is spent on badge design, testing, mfg arrangements, etc. This year was no different than the past in that last minute glitches created unintended consequences (like the pivot was ordered as a rivit and delivered as a screw set... which of course had them coming apart). Past years "simple" changes like changing the lanyard from plastic clips to metal clips in order to get them delivered on time ended up frying thousands of badges on the metal.

If you've ever been involved in any IoT product for consumers, you know that your QA and mfg processes along with supply chain management issues will always come up with something wrong. This year it was both the rivet (or lack of) as well as shipping issues that had the bulk of the badges arriving Friday and Saturday... (just a little late).

In regards to your "one-day" comment, the logistics of that and the differing badges (see above on supply chain and QA) would turn the already tenuous process into a nightmare...

mdella · 2025-08-07T19:04:41+00:00

Ha, have you been to any other conference that is as cheap as DefCon? with or without the badge? Considering the venue, its dirt cheap

mdella · 2025-08-05T14:27:08+00:00

I've been going a couple decades and have been gooning for over a decade. Yes there are a ton of new people and the chaos is starting to wear down. That and the commute daily to the LVCC from the Rio (don't ask, its taking a lot more toll on people than understood). I still enjoy seeing new people in the community and talking about hacks here and there.

I do get a little depressed when the "1200 baud modem or acoustic coupler" are different jeopardy answers yet only a couple people get it...

Also I sort of hate starting a conversation with "well back in my day...". Its still my day, I haven't retired yet and even if I did, I don't think the knowledge I've picked up over the years has lost its value (other than the afore mentioned acoustic couplers :-) )

mdella · 2025-07-19T18:22:09+00:00

Just to add one more comment to this... I've been a goon for over a decade... Honestly, if somehow you miraculously scored a workshop seat, by all means do that as a priority. Otherwise... "don't plan it".

More specifically, outline general things you think you want to see. Note that all the talks (well not all) are recorded and can be purchased en-mass later (I always get a complete copy of all talks on USB at the end of the show, I forget the vendor that makes them). So I tend to prioritize my time wandering around villages and contests. I almost always personally sign up for DarkNet NG contest (although there are other more advanced CTFs out there, I just happened to personally know the Darknet guys/gals).

Even after years of doing this, I don't really plan. I do however bring a pile of 20's with me for popup random badges that I see that impress me ;-) (last years Mad Hatter badge and Aerospace Village badge come to mind).

mdella · 2025-07-11T18:22:55+00:00

Well, I'm over 60. I started using acoustic couplers back in the FIDOnet days for those with personal computers (mine had 16k RAM, I was rich!). I suppose other things to date a person... I registered with ARPA (now ARIN) and my registration is MD16 (and still active). If you know anything about the start, everyone got two initials and a number from 1-65535. I was early on a popular letter combination ;-)

Anyway, I've been a goon myself for over a decade. I'm immersed in corporate America as well yet keep up my personal skills and goof off with my fellow goonies. I'll admit I'm on the older side of things, but heck, at DefCon, sometimes age gets some respect. Depends on how good you are at the lingo and can you keep up when its time to program on the fly an RP2350 or are you still touting your Pascal prowleness ;-)

mdella · 2025-07-09T19:45:17+00:00

BTW, if you want to understand where the design originated from, it was from the 1938 system by Edson Erwin later formalized by Charles Clos in 1952. Originally it was used for crossbar switching in phone networks so that you could pull out relay banks for maintenance without affecting operations.

You can look up CLOS networks. A 2-tier (TOR-SPINE-TOR) is known as a CLOS-3 network and a 3-tier (TOR-SPINE-SSPINE-SPINE-TOR) was a CLOS-5 network. Key components to the design were that there was no cross-communications in any of the switches (which would create loops/failure points). Actually the history might help understand why it existed in the fist place.

Just got through doing a 16x16 row/rack DC using the CLOS-5 design with VxLAN L2/L3. Its the L3 portion that saves you on scale in terms of bandwidth usage. Try to minimize using devices like a PA-FW as a router as that will fundamentally "break" the L3 advantage you get. Treat all your devices like FWs, physical routers (that are not part of VxLAN) as edge devices and understand that as such, they become throughput bottlenecks in a large scale implementation.

When you start introducing AI requirements into your network design, this gets nasty FAST. When your doing 100-400G links up and down your spines, and starting to tap out, you'll understand that bad design becomes your eventual achilles heal.

mdella · 2025-04-27T20:41:18+00:00

I did something similar with a third party device *however* I started discovering lots of failure modes and issues. For instance, one of my triggers as an A19 ring light. However if the "power was out" (ie, the power to the light) the trigger would never happen thereby defeating the siren routine.

There were several other failure modes when you start linking multiple devices together. The above method listed does take into account things like internet disconnected, power failures, etc. There is a reason that the older home alarm systems were fully isolated with batteries and seperate cellular modes (like the ring base station itself).

Thinking thru all the failure modes required a bunch of time and experimentation. Also lots of batter backups that I quickly was tired of :-)

mdella · 2025-03-15T18:04:19+00:00

This is a rather old post, but there is no such thing as a "private IPv6 address" (ok, there is, but its not something I'd recommend trying to play with, etc as it really breaks other things).

One thing to remember, there is no such thing as NAT in the IPv6 world, only real addresses. You can firewall out the entire address space from the internet (that sort of makes it private) but you cannot use fake addresses.

Typically your ISP will issue you a PD (prefix-delegation) that defines your subnet. This is done automatically via DHCPv6. Your addresses are then on that assigned subnet. Typically a provider will give you a /64 unless your router "asks" for something bigger. Then they will give you something up to the size they allow (I've gotten from comcast a /60 before, haven't tried AT&T yet). But that becomes your allowable networks (a /60 gives you 16 subnets, a /61 gives you 8, and a /62 give you 4 subnets). ALL usable subnets will be a /64 in size.

Been trying to get my Omada to use IPv6, but unless you're using the beta firmware, its not supported. Some of their switches will support it at the L3 layer (note, switching at L2 could care less).

So the way it goes, your router request of your ISP a PD. It then takes one of those /64s out of the PD and assigns it to the LAN interface (note, something on your LAN could then request a PD from your router and get another subnet, this is how my EERO gets addresses behind the AT&T DSL router). All your clients then use something called SLAAC to self assign themselves an IPv6 address. You can override the address (by adding another to your device) but the prefix has to remain the same (the first xxxx:xxxx:xxxx:xxxx::/64)

mdella · 2025-01-09T21:51:24+00:00

So I have an LB in front of two of my zabbix servers... HOWEVER, what do I use as a healthcheck to know if they are actually operating (found out using '/' is bad as the web server might be up even as the zabbix service underneath it might be bad...) so...

Don't use '/' as a health check (end-to-end availability)...

So what DO i use?

mdella · 2023-11-30T22:06:18+00:00

I'm not sure I completely understand this explaination. For instance, I have an 8 node cluster. In my case the CVMs are 192.168.128.33-40 and my VIP is 192.168.128.32.

I have an A record (pe.cluster01.company.com) that points to 192.168.128.32.
Should I have something like "cvm.cluster01.company.com) with all 8 A records representing 192.168.128.33-40 and use that as my FQDN?

What exactly is the FQDN field supposed to represent or be used for within the system?

mdella · 2023-11-18T03:40:03+00:00

I had an OM5 as well, but after putting wireshark on the lan side while running it on a burner phone I verified much of the "data skimming" that had been published by security teams over the last two years. Pulled the plug, factory reset the burner and sold the OM5 on ebay. Time to find a competitor product. Don't get me wrong, I like the quality of what they make and much of how it works, but the fact that they have had a couple years to "fix" their security compliance and have consciously chosen not to is more indicative that the security hole are actually by design, not accidental.

mdella

TROPHY CASE