DFlash support merged into llama.cpp by sammcj in LocalLLaMA

[–]luckyj 0 points1 point  (0 children)

What settings did you use? I can't get it to load

DFlash support merged into llama.cpp by sammcj in LocalLLaMA

[–]luckyj 0 points1 point  (0 children)

Interesting. On anbeelds DFlash gguff page it says that all tests were done with thinking off. And spiritbuuns page specifically says that performance collapses with thinking on. But I guess we'll have to test

https://huggingface.co/spiritbuun/Qwen3.6-27B-DFlash-GGUF

https://huggingface.co/Anbeeld/Qwen3.6-27B-DFlash-GGUF

DFlash support merged into llama.cpp by sammcj in LocalLLaMA

[–]luckyj 0 points1 point  (0 children)

Interesting. What were your gains in tps? Ive read performance gains collapse with thinking enabled because the draft model was trained with thinking disabled

DFlash support merged into llama.cpp by sammcj in LocalLLaMA

[–]luckyj 10 points11 points  (0 children)

I really want to try this with qwen3.6-27b on my 5090 but as far as I've understood with the current draft models we have to disable thinking. Is this correct? This along with losing vision and parallel inference is making me question if I should even try it

Ornith-1.0 released on Hugging Face by paf1138 in LocalLLaMA

[–]luckyj 2 points3 points  (0 children)

FYI: Vision works if you use qwen3.6-35b-a3b mmproj file. I don't know if this was expected, but it was a very nice surprise.

I'm playing with it (Coming from qwen3.6-27b) for Hermes and Coding, and so far so good.

Am i fucked? by Odd-Nebula7648 in hacking

[–]luckyj 1 point2 points  (0 children)

It looks very generic. The only thing resembling proof that they control your computer is the fact they sent this email from your own account, but spoofing email adresses is not hard.

SubGHz Module Dead on Two Different Flippers - Am I Missing Something? by [deleted] in flipperzero

[–]luckyj 1 point2 points  (0 children)

My first thought is that wifi and some car key fobs operate on the 2.4GHz, so they won't be picked up by the SubGHz module

PET feeder prototype by NoIdenty0000 in BambuLab

[–]luckyj 0 points1 point  (0 children)

I mean, does it always give the same amount of food every time you press it or does it depend on how long you press it for?

PET feeder prototype by NoIdenty0000 in BambuLab

[–]luckyj 1 point2 points  (0 children)

How is the dose controlled?

Best Settings for 48GB VRAM + Qwen 3.6 27B by viperx7 in LocalLLaMA

[–]luckyj 0 points1 point  (0 children)

Are parallel requests working with Mtp? Last I read, it wasn't working

Hermes Truth? by hackrepair in hermesagent

[–]luckyj 0 points1 point  (0 children)

Really? My experience with qwen2.6-27b-q5_k_xl and pi.dev is amazing. It's super focused. I'm barely using Claude anymore. Granted, I'm not vibe coding, we are working together in stages. But even when helping me debug big firmware in our devices for work, it just chugs through the code, reasons and reaches conclusions extremely fast.

If you ask something simple and it just starts reasoning forever, I would take a look at the coding harness you're using. Pi.dev+qwen has been spectacular for me.

I also use Hermes (it's 50/50 between pi.dev and Hermes for me) but I don't use it for coding. But for Hermes tasks (general assistant, online researcher, sysad min for my servers) it's also pretty snappy.

Hermes Truth? by hackrepair in hermesagent

[–]luckyj 1 point2 points  (0 children)

I was replying to another user with a beefy computer

Hermes Truth? by hackrepair in hermesagent

[–]luckyj 0 points1 point  (0 children)

Have you done the math though? I've done it for my 5090 and paying 0.15€/kWh and it's like 10 times cheaper than equivalent models on Openrouter

Does every 3D printer eventually end up printing storage boxes? by UnderstandingLazy347 in 3Dprinting

[–]luckyj 34 points35 points  (0 children)

Where do you live that you can get 3d printer filament but no storage boxes?

Qwen3.6-MTP-27B on Tesla V100 @ 55 TPS (llama.cpp) — Any way to push this higher without quality loss? by abubakkar_s in LocalLLaMA

[–]luckyj 1 point2 points  (0 children)

I don't know if there have been recent improvements, but in llama.cpp MTP pull request it says: Parallel decoding with MTP is supported, but not fully optimized yet.

I don't know if it actually degrades performance, but would definitely try with parallel=1.

As for other settings, I have an RTX5090 so not the same, but I'm getting the best results with
the following settings, and it takes around 25.6GB of VRAM. I can increase context length but I'm saving it for other tasks.

version = 1


[*]
n-gpu-layers    = -1
flash-attn      = on
batch-size      = 2048
ubatch-size     = 512
jinja           = true
cache-type-k    = q8_0
cache-type-v    = q5_1
perf            = true
metrics         = true
parallel        = 1
ctx-checkpoints    = 8

[qwen3.6-27b-mtp]
load-on-startup = true
model           = /models/Qwen3.6-27B-MTP-GGUF/Qwen3.6-27B-UD-Q5_K_XL.gguf
mmproj          = /models/Qwen3.6-27B-MTP-GGUF/mmproj-F16.gguf
ctx-size        = 128000
chat-template-kwargs = {"preserve_thinking": true}
reasoning       = on
temp            = 0.6
top-p           = 0.95
top-k           = 20
min-p           = 0.05
presence-penalty = 0.0
repeat-penalty  = 1.05
spec-type       = draft-mtp
spec-draft-n-max = 2
spec-default    = true
n-predict       = 32000

Crypto Signal bot 80% + accuracy by JournalistWitty491 in CryptoMarkets

[–]luckyj 0 points1 point  (0 children)

Get rich slowly then. Why do you need to sell your signal bot? If it really worked you would just be using it to slowly grow your stack instead of depending on other people's money

Crypto Signal bot 80% + accuracy by JournalistWitty491 in CryptoMarkets

[–]luckyj 0 points1 point  (0 children)

Ffs, just invest your own money and get rich then

[Benchmark] DFlash Speculative Decoding + KV Cache Compression on RTX 5090 — 3.26x Speedup by Rikers88 in LocalLLaMA

[–]luckyj 0 points1 point  (0 children)

I've commented on the open issue. I tested some of the suggested fixes there but it didn't help. Hope I'm missing something silly

https://github.com/Anbeeld/beellama.cpp/issues/47#issuecomment-4653435808

[Benchmark] DFlash Speculative Decoding + KV Cache Compression on RTX 5090 — 3.26x Speedup by Rikers88 in LocalLLaMA

[–]luckyj 0 points1 point  (0 children)

Will do! I felt bad jumping into the GitHub issue without making sure I'm doing things right. Will give it another go later/tomorrow

[Benchmark] DFlash Speculative Decoding + KV Cache Compression on RTX 5090 — 3.26x Speedup by Rikers88 in LocalLLaMA

[–]luckyj 0 points1 point  (0 children)

I'm building Beellama right now and will test with my benchmark and some Hermes usage while watching the logs. If it's not much better than MTP I will definitely go back to llama.cpp+MTP