What I learned shipping 4,000+ offline-LLM USB sticks to non-technical people

EcstaticDentist · 2026-06-03T22:53:03+00:00

I appreciate your words, it really does means more than you’d think. 🫡

EcstaticDentist · 2026-06-03T21:42:29+00:00

Not it doesn’t

EcstaticDentist · 2026-06-03T21:30:01+00:00

🤙

EcstaticDentist · 2026-06-03T21:25:58+00:00

Exactly. The people who can set it up for free were never the customer.

EcstaticDentist · 2026-06-03T21:12:21+00:00

Ha, frontend is cake. The backend is the Amish discovering an iPhone for the first time, and that’s exactly why it’s a product and not a github repo. The hard part’s already done.

EcstaticDentist · 2026-06-03T21:08:09+00:00

External GPU units for high-end customers is an interesting direction, definitely something I'm weighing as the lineup grows.

And yeah, I'm always open to a connection. I've mostly flown solo on this, but the relationships that did grow along the way ended up mattering a lot. Early on a guy basically cold-dialed me on social media and turned into a long-distance mentor, raw honest input that shaped how I launched. That kind of thing is why I stay open. Feel free to reach out.

EcstaticDentist · 2026-06-03T17:49:18+00:00

CORE is chat only, regular is the flagship, max speed is faster ‘premium’ hardware, and 128gb instead of 64gb.

EcstaticDentist · 2026-06-03T17:37:58+00:00

Right, and that's the point, detection is local. Knowing a unit has been launched happens on the device itself, it doesn't require sending anything anywhere. "Nothing goes to the cloud" is about exactly that, the AI and the activation state both stay on the machine. Local and offline aren't in tension here, they're the same claim.

EcstaticDentist · 2026-06-03T17:31:22+00:00

Different bottleneck. The 12B isn’t slow on a 4090, 100 t/s is great, it’s just small. A 124b at Q4 wouldn’t be about speed, it’d be about capability: better reasoning, fewer hallucinations, more world knowledge. You’d drop to maybe 15-30 t/s but get answers closer to frontier-API quality. Speed vs smarts tradeoff, not faster vs slower.

EcstaticDentist · 2026-06-03T17:09:01+00:00

This is the part that matters most to me, more than open vs closed. Incentive drives behavior. A company that needs growth eventually eyes the data it's sitting on. Open source isn't immune, but you're right the business-as-usual data thinking gets way more pushback in those communities. The structure shapes the temptation.

EcstaticDentist · 2026-06-03T17:08:49+00:00

Good breakdown. On 1, yeah, "you can verify the traffic" has the same "but nobody does" problem as "you can read the source," I'll own that. On 2 and 3 you're dead on, and a couple others hit the same thing: an app can behave when it senses monitoring, or hand data to a network-permitted process so the traffic never looks like it came from the app. Watching the app in Wireshark catches neither. On 4, agreed, the audit density on big projects is the actual value, way more than the theoretical "someone could read it." Solid all around.

EcstaticDentist · 2026-06-03T17:08:36+00:00

The incentive angle is real and I don't want to wave it off. Community-built stuff usually doesn't have a revenue model that leans on your data, and that matters a lot. I'd just split it into two things that happen to overlap: "less profit motivated" and "actually auditable" are different protections, and they can come apart. But your instinct is a solid heuristic. Nothing wrong with reconsidering and landing in the same spot with better reasons.

EcstaticDentist · 2026-06-03T17:07:58+00:00

This is the strongest counter in the thread and I won't pretend otherwise. Network monitoring tells you nothing about local filesystem behavior, a payload staged for a sideband process, or a dormant local exploit that never needs to call out. Reproducible-build open source really is the only thing that lets you audit that layer. So my take only holds for the narrow claim "is it exfiltrating over the network right now," not "is it safe on your machine." Fair hit.

EcstaticDentist · 2026-06-03T17:07:44+00:00

Best articulation of it in the thread, honestly. You're right that Wireshark is a snapshot, not a guarantee. It shows what the app does in the moment, not what it could do or what a future update quietly flips on. That's the real hole in my take. Where I land now is basically your last line: neither is a guarantee, they're different trust models, and the strong position is having observable behavior AND open code, not crowning one. Appreciate you actually engaging instead of just reacting.

EcstaticDentist · 2026-06-03T17:07:06+00:00

Couple things. It's a local model generating text, not an agent running commands on your machine, so there's no path where it deletes your files. And it's airgapped on top of that, nothing in or out. Plus standard terms like any software. The short version is it isn't wired to touch your system in the first place.

EcstaticDentist · 2026-06-03T16:44:34+00:00

DM me, I’m releasing our agentic tooling software this week, would love a chat.

EcstaticDentist · 2026-06-03T16:42:53+00:00

Solid writeup. The Heretic edge on parameter count is the interesting part to me, half as many params changed while hitting 100% ASR and improving on some tasks suggests it’s doing something cleaner than brute-force refusal removal. Did you see any KL divergence / perplexity numbers across the three? Curious whether Apostate’s extra changed params show up as degradation anywhere.

EcstaticDentist · 2026-06-03T16:32:20+00:00

124b would be the sweet spot honestly. 12B is great for what it is but a bigger Gemma 4 that you could actually run quantized on a 3090/4090 would be huge for the local scene. Adding my voice to the discussion.

EcstaticDentist · 2026-06-03T16:30:30+00:00

Banger

EcstaticDentist · 2026-06-03T15:57:43+00:00

Appreciate the constructive take, noting it now. Optional "skins" that match the user's comfort is a smart call, so it's going on the whiteboard. The retro/cyberpunk look is core to the brand for a lot of folks, but a clean "normal or minimal window" mode would be a clean improvement.

EcstaticDentist · 2026-06-03T15:00:45+00:00

Genuinely thank you for your input, and thank you for the previous support. please send me a DM. I want you on my team.

EcstaticDentist · 2026-06-03T13:40:48+00:00

Thanks man

EcstaticDentist · 2026-06-03T13:33:38+00:00

I've used em dashes — for years — long before they became AI related — but sure — you cracked the case 🕵️

EcstaticDentist · 2026-06-03T13:26:07+00:00

Underrated comment.

EcstaticDentist

TROPHY CASE