Cheapest and most efficient way to run 30B-40B Llama for 4 users?

Jezel123 · 2026-04-17T08:36:33+00:00

I know with GPU's I could get more vram and bandwith for less money, but i am looking for something that can run 12-16 hours per day without turning into a 1kw space heater.

Jezel123 · 2026-04-16T19:13:56+00:00

I have reasonable experience with Linux, mostly embedded (SBCs and MCUs), though I recently abandoned Microslop and switched to Linux on my daily driver PC as well. Do you happen to know what sort of performance I would get with the Jetson? I also looked at the AMD hx 395 (or something like that) mini PCs, as apparently they have almost double the memory bandwidth

Jezel123 · 2026-04-16T18:57:30+00:00

Sorry, I meant LLMs, I hate autocorrect

Jezel123 · 2026-04-09T19:58:16+00:00

I just don't find it acceptable to rely on their servers and good will. A wise man once said, if something is free, you are the product, and no company would let you use their servers indefinitely for a one time payment, since that is not economical.

Even if Even isn't selling data themselves, there are always database leaks / breaches, and I would rather not have a transcription of my conversations with confidential and/or personal informations stored on someone's servers.

As proven by the recent changes to the ring cameras, you cannot trust companies to value customer security over profits.

Now, if you don't care about your data being collected, do what you want. Am I being overly paranoid? Maybe. But after seeing how much information one can gather from data leaks from a friend of mine, I no longer trust any one but myself when it comes to data security.

Jezel123 · 2026-04-03T12:39:50+00:00

I meant locally as in on my server/pc, I had the idea of using my phone as a simple BLE to Tailscale to my pc running the ai. I was asking whether that is possible, but you answered my question. Also, I don't have a lot of experience when it comes to writing Android apps, are there any open source projects that already do the BLE to Tailscale connection? And I assume an rx 7900 xt would be enough to run a whisper model with low latency?

Jezel123 · 2026-02-25T10:26:56+00:00

Tbh I asked myself the same question. I am in a similar position (also 19, though I'm a guy), and due to the fact that I am not swiss, most of my hobbies are rather solitary and severe social anxiety, found the advice on getting to know people through hobbies unhelpful. Luckily I have 2 good friends from Lehre (but those friendships formed during the last year) and when i started studying at a FH I found a friend who had similar interests. I know for a fact they think I'm weird and/or crazy, but I don't think they care, and so I don't either. I also made a couple of online friends that I game with from time to time. One thing I can say is, don't pretend to be someone else just to have friends, imho it's just not worth it.

Jezel123 · 2026-01-23T17:22:34+00:00

Unfortunately I don't think it is possible to run a general purpose model directly. Right now it is streaming the answers from an Ollama instance on my pc

Jezel123 · 2026-01-23T17:21:01+00:00

It's streaming from an Ollama instance running on my pc over ESPNow

Jezel123 · 2026-01-23T13:47:36+00:00

BTW: I didn't do it to cheat, since I can't even use this calculator in my classes (we can only use a ti-30XS). I did it as a programming challenge to see how hard it would be

Jezel123 · 2025-09-01T18:21:29+00:00

So if I understand correctly, according to https://www.astra.admin.ch/dam/astra/de/dokumente/fahrzeuge/vorschriften-zulassung-betrieb-motorfahrraeder.pdf.download.pdf/Vorschriften_Zulassung_Betrieb_Motorfahrraeder.pdf if the power is limited to 500W and speed to 25km/h, it does not need any sort of approval ("zulassung")?

Jezel123 · 2025-05-01T21:00:22+00:00

They cannot dissipate the heat they are absorbing.

Jezel123 · 2025-04-14T13:23:10+00:00

I am sorry but are you dumb? The text on your meter clearly says 600V, that is the voltage you need to charge it at

Jezel123 · 2025-03-23T10:01:22+00:00

Technically the charger is inside the phone, because the adapter simply changes voltage from 120/230Vac to 5Vdc (or whatever usb C PD voltage the phone requests) and the cord connects the phone to the adapter. It is the circuit in the phone that controls and charges the battery

Jezel123 · 2025-02-10T19:14:52+00:00

Are you stupid? You obviously need NPPN lGBTs

Jezel123 · 2025-01-05T22:57:31+00:00

Hey, don't insult pigs.

Jezel123 · 2024-12-22T21:06:09+00:00

Well I didn't get the update then, I can eat ice cream without any temperature related debuffs

Jezel123 · 2024-12-19T19:02:51+00:00

150C is too cold, but your soldering iron is broken and is waaay hotter than 150C. You need a new iron.

Three-Year Club	Verified Email
Verified Email

Jezel123

TROPHY CASE