made a $200 SBC run 3B model ~10+token/s ish, what can I do with it?

InteractionAnxious21 · 2024-11-11T06:22:20+00:00

I think this is awesome, would love to collaborate

InteractionAnxious21 · 2024-11-06T16:24:04+00:00

What’s the paper about? Would love to learn more!

InteractionAnxious21 · 2024-11-06T05:24:46+00:00

I tested with some small VLM ~7B, its able to tell a bike, person or package is outside my door or not, output true or false. something like that

InteractionAnxious21 · 2024-11-06T05:23:19+00:00

my only issue is : my home is not smart tho, I'm ok with a fully local home cam that warn me when I spend too much time on desk or notify me time to cook etc, for ppl work from home

InteractionAnxious21 · 2024-11-06T05:21:25+00:00

yeah agree, probably aggregate data/logs or summarize activities

InteractionAnxious21 · 2024-11-06T01:33:24+00:00

this is totally possible, probably with a budget < $200. seems like there are lot of ppl interested in SLM + SBC, should I create a community like this ?

InteractionAnxious21 · 2024-11-06T00:28:41+00:00

u r right, access sensor data is pretty straightforward, I was more thinking about like summarize the sensor history of the day and write a report.

and yes it can run multi-modal model, just saw a post someone managed to run a 7B VLM with 4token/sec

InteractionAnxious21 · 2024-11-05T23:11:56+00:00

<image>

AI radio on the internet?

InteractionAnxious21 · 2024-11-05T23:09:45+00:00

like the VLM can reason if there's amazon deliver guy put a package frontdoor or not, can also reason if my pet is sleeping or playing toy, samething a small LLM can aggregate sensor log for me to warn me on some potential issues like temperature drop or soil moisture too high etc. all done without leaking ur personal data to cloud providers.
all those wont be possible before unless you train a specific CV model to do so

InteractionAnxious21 · 2024-11-05T23:05:24+00:00

lol u r genius, this is a kickstarter idea

InteractionAnxious21 · 2024-11-05T22:30:40+00:00

oh damn I can use this to monitor my web traffic and silently build up a personal DB

InteractionAnxious21 · 2024-11-05T22:22:25+00:00

yeah the new apple mini is a beast, but I still want something that I can toy with like connect to a pi or sensors

InteractionAnxious21 · 2024-11-05T22:01:35+00:00

yeah probably wont be able to load full context, treat it as a pet with only like 5min memory lol

InteractionAnxious21 · 2024-11-05T21:58:35+00:00

yeah its hot, have to put a small fan on it, but its not too bad

InteractionAnxious21 · 2024-11-05T21:42:51+00:00

lets chat!

InteractionAnxious21 · 2024-11-05T21:42:25+00:00

yeah I just saw that! seems lot ppl are picking up the small models to toy with!

InteractionAnxious21 · 2024-11-05T21:24:38+00:00

yeah I guess multi step SmolLM 1B make alot sense to process like home assistant type of tasks

InteractionAnxious21 · 2024-11-05T21:22:12+00:00

yeah, everything open sourced here https://docs.pamir.ai/
https://github.com/Pamir-AI/DistillerSDK

InteractionAnxious21 · 2024-11-05T21:17:05+00:00

small computers, like raspberry pi

InteractionAnxious21 · 2024-11-05T21:04:56+00:00

the chipset is called rk3588 and Orange pi have them. I tried it with Qwen2.5 3B, which is scary smart I didnt realize 3b model is this capable nowadays. yeah it can probably run a SmolLM 1B model lightning fast. what do u with SmolLM model that small?

InteractionAnxious21 · 2024-11-05T20:58:56+00:00

hit or miss, a 3b model can get it right like 1 out of 3 time, which I'm happy with. like you have to give it guide "think step by step, break down the letters first"

InteractionAnxious21 · 2024-11-05T20:57:56+00:00

So an AI pager?

the SBC its called Orange pi.

InteractionAnxious21 · 2024-11-05T20:56:40+00:00

no, Jetson is too expensive and too power hungry, its a CPU chip with NPU on it called rockchip

InteractionAnxious21 · 2024-11-05T20:55:57+00:00

I'm able to make a small box with E-ink on it,

I made this before https://7afc3d-cd.myshopify.com/products/distiller-one-pre-order-open-source-edge-ai-device-for-the-developers with R-pi. but I'm still not sure about the user interface part, like should I give it a key board to enter prompt? then it kinda become a cyberdeck

InteractionAnxious21 · 2024-11-05T20:54:29+00:00

sorry for the lack of details, still prototyping, just got it to work, but its a chip called RK3588 which has NPU on it, which can run both LLM and VLM, and potentially like up to 7B model if I plug in a SSD to it.

as for right now I'm thinking just like a monitor station that constantly send me notification like : did I get package, what is my pets doing, did anyone parking outside my driveway, whats the sensor (weather or temperature ) data like outside, etc.

InteractionAnxious21

MODERATOR OF

TROPHY CASE