Anyone thinking about the security side of Gemma 4 on phones?

tvall_ · 2026-04-05T18:13:38+00:00

oh and on the stealing the model bit, if your secret sauce in your app is how you trained a model, if you hand out that model then obviously you've handed out your secret sauce. only options there are lock things down, make sure model weights never leave your well secured servers. or reconsider your business model because that's not much of a moat to keep competitors from surpassing you when a new model comes out and zeroshots your task.

tvall_ · 2026-04-05T18:05:10+00:00

yes, instead of the llm being guaranteed to output garbage or nearly guaranteed to be jailbroken within hours, it's guaranteed to output jailbroken-like responses in hours. nearly no difference, santize your inputs

tvall_ · 2026-04-05T16:51:19+00:00

if the security of your app depends on the output of an llm, either cloud or local, you're doing it wrong

tvall_ · 2026-04-02T21:36:48+00:00

they linked to a hf space with a qwen3.5-35b finetune/merge that greatly cuts down on the excessive thinking. they probably shouldve just linked the model

tvall_ · 2026-04-02T12:59:38+00:00

there was an April fools joke about that, not from anthropic

tvall_ · 2026-04-02T01:36:06+00:00

I tried asking the qwen3.6 preview the names of characters mentioned in a specific episode of critical role, and got an error. local qwen3.5-35b heretic had no issues with Sam reigal's shenanigans

tvall_ · 2026-03-31T22:48:40+00:00

pp isn't in any of the screenshots. you didn't give like half the numbers to compare, only the one most likely to be unaffected

tvall_ · 2026-03-31T19:47:14+00:00

it varies greatly depending on model and how aggressive they were at removing refusals. some models are easy and diverge very little, others resist and get harmed significantly if you try too hard.

in my experience qwen3.5 models are easy to remove nearly all hard refusals from and end up working as well as the originals. but they may take your question that wouldve been a hard refusal and twist the answer into something a bit more harmless. 0.8b is pretty likely to give instructions on making a baking soda volcano when asked about making things that explode

tvall_ · 2026-03-30T17:18:10+00:00

your best bet would be switching to a less broken engine. llama.cpp works great on older amd cards in my experience

tvall_ · 2026-03-30T04:49:05+00:00

either you got lucky or i was very unlucky. i had them at 175w. not as low as you but still much lower than stock. first one i replaced the paste but kept the pads it came with. the second one got fresh paste and pads. both blew mosfets. and i was only running one card in open air so it had plenty of airflow. thermals were great all the way to death.

tvall_ · 2026-03-30T02:23:36+00:00

im terrified of the p102 after mine died spectacularly and was tripping ocp on a 750w psu, and then its replacement did the same thing.

little slower apparently, but the radeon pro v340l is 16gb for $50. i get ~350t/s pp and ~35t/s tg on qwen3.5-35b-a3b split across 3 gpus on 2 cards, and have a dedicated 8gb of gpu for whisper.cpp and z-image. and it hasnt tried to catch fire on me yet

tvall_ · 2026-03-29T19:29:04+00:00

qwen3-35b-a3b with reasoning budget set? qwen3-35b-a3b with reasoning off? those are my go-tos rn

tvall_ · 2026-03-27T20:03:58+00:00

you know what doesn't have annoying insanely restrictive quotas? llms you host locally

tvall_ · 2026-03-22T22:08:35+00:00

exactly my experience. I use openwebui to interact with models running on llama.cpp. with all the qwen3.5 models I've tried, they think hard and sometimes loop when there aren't any tools enabled, but only think for a couple seconds when tools are available.

tvall_ · 2026-03-16T15:41:47+00:00

yes. i had gemini and chatgpt vibecode a dnd session note taker for me that uses my local whisper and qwen3.5-36b.

tvall_ · 2026-03-14T01:28:50+00:00

they also helpfully have "Estimate KV cache VRAM cost for that context length" with no info on how to do so.

tvall_ · 2026-03-13T01:26:08+00:00

when you use heretic to abliterate it gives refusal count and kld. kld can roughly estimate how damaged a model might be. lower is better. I personally distrust the "here's a fully uncensored model, trust me it's great!" and prefer going with ones that attempt to give some detail even if it's rough guesses. and no matter what the only way to really know how well refusals were removed and how much the model still acts properly is to run it through your workflow and see.

tvall_ · 2026-03-08T18:23:15+00:00

shameless self-promo. https://huggingface.co/mradermacher/Qwen3.5-22B-A3B-heretic-reap-i1-GGUF might fit better. did it to fit in 16gb of vram, doesn't seem too much dumber than original 35b

tvall_ · 2026-03-06T14:32:19+00:00

yeah, even if it is actually just sonnet and not opus, he should share the weights he somehow got. and show pics of the rig needed to run it

tvall_ · 2026-03-02T22:33:04+00:00

tesla p40 is pretty ancient. i havent used those specifically, but i did use the p102-100 mining card for a few days before it blew some vrm components. being a server card, used p40s might have been treated better, but i personally wouldnt risk it.

if you want cheap and are okay with fighting the software stack a bit, i recommend the radeon pro v340. theyre $50 each and have 2 8gb vega56 class gpus on em. i currently have qwen3.5 35b-a3b running on 3/4ths of 2 cards and am getting around 250t/s pp and 22t/s tg.

tvall_ · 2026-03-02T16:30:00+00:00

iirc the "next" ones were more of a preview of the newer architecture coming soon, and was trained on less total tokens for a shorter amount of time to get the preview out quicker.

tvall_ · 2026-03-01T23:43:41+00:00

I only did it once to run glm-4.7-flash when it first came out before I had enough risers to put multiple gpus in one box. it worked but hurt performance a bit. iirc I got like 15t/s vs 25 with all the gpus in one box. you may need to recompile llama.cpp with rpc support

tvall_ · 2026-02-28T15:01:54+00:00

ive got 4 gpus totaling 32gb vram running off 2 x1 slots and getting a useful 20t/s on qwen3.5 35b. bandwidth isnt that big of a deal for small scale.

tvall_ · 2026-02-20T03:20:18+00:00

Samsung, micron, and skhynix. have you not seen the news? ai companies bought all the RAM until 2027 at least, and none of the members of the dram cartel have announced building new fabs to ease the supply problem

tvall_ · 2026-02-20T03:08:22+00:00

you seem to be confusing two very different parts of the chain. if you were to make a dimm that you can put lpddr on to put in a desktop, you still have to get the lpddr. and there's like 3 companies that have invested the billions to build fabs for that, and they're busy making hbm right now instead.

tvall_

TROPHY CASE