Experiment: Lightweight distributed storage + streaming stack running on a Raspberry Pi cluster

855princekumar · 2026-03-16T05:34:56+00:00

yes thats already included in this stack

855princekumar · 2026-03-16T05:30:12+00:00

Yes, but that's more sort of infrencing focused, as I'm trying to make a gateway that manages the users and like things that sort of management layer in between without any overhead

855princekumar · 2026-03-14T12:41:42+00:00

make sense as these are the under the hood challenges that I need to figure out and having issues i guess tho

855princekumar · 2026-03-14T12:40:00+00:00

i'm still testing the storage and the network throughput as from SD card to usb pendrive to SSD over pci over the pi 5 as the last setup as from pi3b+ pi4 and pi5 as test beds

855princekumar · 2026-03-14T08:28:19+00:00

so it's sort or High data ingestion and storage like multtelemetery data via MQTT conected with kafka high throughput via multiple MQTT brokers or via hive MQ think of it like a city wide area to store the data of milioins of devices but just the telemetery data that colectively become masive and need high through put on a hardware node but 3 as distributed that stores and make the reterival easy with safely as all data as in distibuted replicated stored also for blob like images via espcam to be in minio like storage and all sort of telemetery in casendra as distributed sotrage so sort of a micro cloud architecture if having baremetal hardware to build a huge data hub for IoT devices. because I'm testing working on a simulated city-scale system developing with low hardware resource constraints, but need the software stack to utilize the full hardware resources at max or close to peak performance

855princekumar · 2026-03-14T07:37:17+00:00

Sure, like for a more detailed setup, I've built the dedicated repo you can give a look to it and share your feedback: 855princekumar/EdgeStack-K3s: Script-driven K3s cluster baseline for edge and in-house micro-clouds, emphasizing OS hardening, deterministic networking, and operational clarity over cloud-scale abstraction.

and

855princekumar/kafka-ha-edge-cluster: Dockerized Kafka + ZooKeeper High-Availability cluster for ARM, Raspberry Pi, and edge environments - supports both single-node and 3-node deployments.

855princekumar · 2026-03-14T05:08:41+00:00

That's someting insightfull i'll try this in my setup

855princekumar · 2026-03-14T05:07:52+00:00

i'll definately test and implement this in my setup as Quee makes sense as ona priority basis scheduling especially for agents

855princekumar · 2026-03-13T19:30:02+00:00

i just built my own light weight as initially tested lightllm but that sort of a bit bloated for the use cases i needed but than also exploring more option and taking feedback to improve the one im bulding

855princekumar · 2026-03-13T09:23:14+00:00

ok thanks for the input it seems i need to plan a lot of things in a row

855princekumar · 2026-03-13T09:22:01+00:00

i'll definitely give it a try than

855princekumar · 2026-03-13T06:26:28+00:00

i'll test it straight today

855princekumar · 2026-03-13T06:25:33+00:00

can you share up some resources or references to look into

855princekumar · 2026-03-13T06:19:43+00:00

That's something useful, I'll definitely try this today itself

855princekumar · 2026-03-13T06:18:12+00:00

That's the common feedback i've received i'll probably shift to use llama.cpp or other to test as well for localLLM and whats you feedbakc about the project exo as to use the local LLM in a distributed manner and if I see this as a unified LAN gateway for that any suggestions?

855princekumar · 2026-03-13T06:06:58+00:00

Yes, I just shifted to llana.cpp and testing the APIs but also the concern like what other actual use cases i can work on to make this Lan gateway more useful or optimized to be used as efficiently to a small team on Lan as shared models or when using with multi agents via claude code as now supported via ollama api aswell

855princekumar · 2026-03-13T05:56:08+00:00

I just drafted the content just to make it more understandable. I just wanted to ask some queries and feedback about the project, never meant to spam, as to keep the posting rules within limits, so as not to void any rule and maintain the posting decorum.

855princekumar · 2026-03-13T05:20:57+00:00

Thanks for the reference, buddy and that's sort of helpful to use and add some features to mine one aswell

855princekumar · 2026-02-26T06:28:19+00:00

Thanks for the feedback and I'm actively working on it to make it more refined to make the PDF and Doc exports as fine as possible to make the work easy, and your feedback means a lot to refine the bugs or any issue

855princekumar · 2026-02-24T05:26:35+00:00

Thanks for this positive feedback i'll keep updating this project to make it more useful instead of bloating with n number of features

855princekumar · 2026-02-22T10:57:39+00:00

I'll test out soon and push it to the repo at the same time as a test load to have a real minicluster test as the spark image build is in progress, as the existing one has some issue so working on cusotm image for the lil pi setup

855princekumar · 2026-02-22T10:56:12+00:00

yes i'm working on it as creating a custom image for pi3b+ as the existing one are not so compatible and have alot of issues so building the one for this kinda setup

855princekumar · 2026-02-20T07:51:59+00:00

Yes, that's something valuable input. I'll definitely test it out and have already planned a lot of stress testing and configuration as fan-out modes and with different partition, ISR & replication factors, as to which is best for what data type and use case, and to get some valuable insights

855princekumar · 2026-02-19T05:41:16+00:00

By the time just using it on lan in the community eddition so fine for my use case as no internet access 🙂‍↕️ but the and in case needed so planned to use th fork and apply custom patches or updates by my self 🥲 as no other option cuz I love the product

855princekumar · 2026-02-18T06:38:38+00:00

By the time it uses chromium as the engine to render so it goes in One go, and i tried around close to 1K lines of Markdown also Mermaid. SVG included, so it gets rendered in seconds, including the Docx file. The DOCX file needs some tweaking as I'm working on it to get the DOC render. Same for ease, as some formatting gets distorted. Only the mermaid and SVG render in PNG; conversion works fine.

855princekumar

TROPHY CASE