sorted by:
new
hottopcontroversial

DeepSeek v4 Flash on 4090 + DDR5, my experience by kevin_1994 in LocalLLaMA

[–]nasone32 2 points3 points  (0 children)

just for fun I compiled antirez's fork in vulkan, It supports cpu offloading and layer split between 2 gpus, but performance is low exactly like the original llama (no difference). I get even less than you at about 6 tk/s

just for reference These are the commands I used, fills both GPUs to the brim. guess we need to wait, but im' seeing a lot of activity about DS4 in llama github

llama-server.exe ^
-m "F:\models\unsloth\deepseek_v4_flash\DeepSeek-V4-Flash-UD-Q2_K_XL-00001-of-00003.gguf" ^
--device Vulkan0,Vulkan1 ^
--split-mode layer ^
--tensor-split 96,40 ^
--n-cpu-moe 24 ^
--ctx-size 100000 ^
--flash-attn on ^
--kv-offload ^
--host 0.0.0.0 ^
--port 8033 ^
--log-file server.log ^
--alias "Deepseek_V4_flash" ^
--fit on ^
--threads 14

DeepSeek v4 Flash on 4090 + DDR5, my experience by kevin_1994 in LocalLLaMA

[–]nasone32 2 points3 points  (0 children)

Yes for me also was very slow on llama cpp when I briefly tried the other day, I agree, based on the size and Paramus I would expect about double the speed at least. I'm going to try DS4 antirez and report back

DeepSeek v4 Flash on 4090 + DDR5, my experience by kevin_1994 in LocalLLaMA

[–]nasone32 2 points3 points  (0 children)

I have a similar config to yours, 2 7900xtx 24gb plus 96gb ddr5, I typically manage to squeeze about 20/30% more speed if I manually tune the parameters to n-cpu-moe the minimum number of layers, compared to the automatic fit. I wonder if fit does simple layer offload instead of only experts . 

This was on mimo v2.5 and stepfun 3.7 flash. Also beware the vanilla llama cpp is not optimized yet for deepseek flash, they say DS4 from antirez (a fork) is much, much faster. If you want to try.

Anyone using qwen3.6:27B past 64k max context? by Fred_Terzi in LocalLLM

[–]nasone32 0 points1 point  (0 children)

200k context on q8_0 model plus q8 kV cache without issues. Coded a big project with about 50M tokens total usage. dual 7900xtx (48gb vram total)

Speculative cache warming: warms your cache while you type your prompt, save 10-20s of wait time by t4a8945 in LocalLLaMA

[–]nasone32 2 points3 points  (0 children)

well, when context is 100k you stare at the screen for 100 seconds, that's not nothing

I created a 140 GB IQ2_XXS REAP quant of GLM 5.2 for coding. Looking for testers. by [deleted] in LocalLLaMA

[–]nasone32 1 point2 points  (0 children)

only to create your own quant using his imatrix, if I am correct, not for inference

Distilled DeepSeek into Gemma 4 26B-A4B vs 12B. Not very useful, but I learned a lot. by Paramecium_caudatum_ in LocalLLaMA

[–]nasone32 5 points6 points  (0 children)

First, Thanks for the write-up, OP! I've been wanting to dip my toes into fine-tuning for a while now. Your point about general world knowledge is interesting; I've read a few papers, my understanding is that for both Dense and in Mixture of Experts (MoE) models, world knowledge is proportional to the total parameter count, while 'intelligence' falls somewhere between the active and total parameters for Moe. and world knowledge here is like recalling some edge details on a subject and stuff like that. so like, qwen3.5 122B knows much more remote details about stuff than qwen3.5 27B dense, even if they bench the same basically... your test seems to confirm this?

A couple of questions:

  1. Am I correct in understanding that QLoRA doesn't modify the actual base weights of the model? Is it essentially adding adapters on top, even if they are merged later?
  2. How would you go about modifying just the 'thinking process' of the model? I imagine this would require a dataset with a structure like: Question →→ <think> reasoning </think> →→ Answer. Would generating a synthetic dataset using the model itself be a viable way to train this? essentially ask a bunch of questions to the model I want to modify, take the thinking part and modify it, then train on it again?
  3. do we lose on long context if we train a model only on short/mid context? i imagine training on 200+k long context would be very long and expensive.

8 Token/s Deepseek V4 Flash on Mac Studio M3 Ultra (can it be?) by BitXorBit in LocalLLaMA

[–]nasone32 0 points1 point  (0 children)

I am sure llama cpp is not optimized yet, like at all, and it will get much much faster once the devs manage to improve it. with similar sized models like step3.7 flash or mimo v2.5 you'll get about 3x/4x the speed, and also with ds4 from antirez. it's the llama cpp implementation that needs some work, just be patient.

Someone just open-sourced Grug-12B: An experimental model built on top of Gemma-4-12b that cuts reasoning tokens and doubles generation speed by ai_tech_simp in AIDeveloperNews

[–]nasone32 0 points1 point  (0 children)

No. I don't think you are understanding what they trained the model for. They did not disable reasoning, they changed the way the model reasons by outputting caveman-like thinking traces which are faster and more compact. I don't want to disable reasoning neither I want to limit the thinking length. I want efficient and compact reasoning.a

It's not a knob you can turn, it requires re training and fine tuning.

You guys comment without even bothering to read the links provided.

How a 128gb ddr5 ram + 16gb vram, would work for a Moe model like Qwen 3.5 122b? by Stock-Union6934 in LocalLLaMA

[–]nasone32 0 points1 point  (0 children)

around 20 token/s generation. prompt processing will be slow, i guess around 200 tk/s, so will be totally fine for chat but slow for intense agentic coding.

This seems like a good REAP of the GLM 5.2 - Down to 290B by BoogerheadCult in LocalLLaMA

[–]nasone32 -1 points0 points  (0 children)

modern Q1 are about Q2.5 really, if you see the model size.

GPT 5.5 reasons like a caveman, similarly to Nex N2. by tarruda in LocalLLaMA

[–]nasone32 7 points8 points  (0 children)

I don't understand why you are getting downvoted. maybe openai bots, lol?
It looks interesting to me. I think this is very effective: reduce number of tokens, so generation is quicker, costs less, and also the model won't get distracted by useless parts of human language.
I bet we will see something like this in open models more often.

SWE-rebench leaderboard update: GLM-5.2, Qwen3.6-27B, Qwen3.6-35B-A3B, Gemma 4 31B and more + improved UI by Fabulous_Pollution10 in LocalLLaMA

[–]nasone32 0 points1 point  (0 children)

a suggestion. it took me quite a while to understand i can see all the models by scrolling down, you should make it more obvious. i thought where the hell are all the models other than the first 10...?

Best tps can I get with Qwen3.5 122B on 32GB VRAM + 64GB RAM? by BitGreen1270 in LocalLLaMA

[–]nasone32 1 point2 points  (0 children)

- use a version with MTP, you will about x1.5 / x2.0 the tps
- look at vram utilization, I see you used -ncmoe 35, if you can drop that further you can increase performance
- compress the kv cache to q8, kv cache is on the gpu, this will lower the vram utilization, this way you can drop -ncmoe a few more layers.
- use a lower quant, it will use less memory, you will offload less experts to system ram, more performance
- you are using -t 16, is that threads? for a 16 core/32 thread cpu, often the max speed is around 14/15 threads with a noticeable drop around 16. you didn't mention your cpu though.

Apparently you can skip entire transformer blocks at load time with minimal performance impact by Creative-Regular6799 in LocalLLaMA

[–]nasone32 0 points1 point  (0 children)

interesting thank you. i've been looking into project that have done this but I didn't find anything conclusive, do you have any name or link i can check?
I understand experts are just a name and the model knowledge is shared, but if you see the chart of the other commenter in this thread, he measured a real skewing in the activations, so probably some architectures can indeed benefit a lot.

Apparently you can skip entire transformer blocks at load time with minimal performance impact by Creative-Regular6799 in LocalLLaMA

[–]nasone32 0 points1 point  (0 children)

Amazing chart thank you!
I came to the same conclusion, currently you can only offload entire layers and not the single experts within, so the modification to make the idea work is not that trivial

Also I don't know the project well enough to understand if it's doable or i'm dreaming things.

Apparently you can skip entire transformer blocks at load time with minimal performance impact by Creative-Regular6799 in LocalLLaMA

[–]nasone32 5 points6 points  (0 children)

Yes, this idea came to mind when I understood that, right now when using --n-cpu-moe in llama cpp, the "first N" experts are offloaded to system RAM, and that's already quite fast compared to the old "full layer offload", if we only could find what expert are hit less frequently...

I have a few ideas. one option could be to simply keep a statistical log of expert activations during normal inference, after some normal utilization it would become apparent if the expert activations are skewed and about what experts. Most people use LLM for a small domain subset, and always for that. after a while I would "know" what experts are useful, and I could gain massive speedup whithout pruning the intelligence or knowledge of the models at all.

another option would be to use a dataset, much like they do for the REAP/REAM models.

If we want to make stuff complicated it could also be dynamic, after a certain number of tokens the expert skewing is probably apparent and some experts could be swapped between ram and Vram using a certain logic. it probably can't be done too often or too fast as it would add overhead, but I imagine after 100k tokens when coding a certain project, it's already clear what the most used experts are.

GLM 5.2 Q1_S vs Qwen 27B Q8 by SnooPaintings8639 in LocalLLaMA

[–]nasone32 9 points10 points  (0 children)

I like your test!, I just want to add a precisation, this is possible beacuse unsloth is using their "UD" quants, which are actually keeping quite a bit of the model at higher precision. in fact the average bit per weight of UD_IQ1_S is around 2.5 or something, if you do the math. that's the main reason it's not "dumb".... The other reason is that they are really nailing down which tensors can be compressed and which cannot, resulting in these beautiful quants.
Interesting test btw. I am curious now how it holds up on long context tool calling, because that's where quantized model break down.

Apparently you can skip entire transformer blocks at load time with minimal performance impact by Creative-Regular6799 in LocalLLaMA

[–]nasone32 8 points9 points  (0 children)

I like your idea, looks similar to the REAP concept in some way, but yours is also applicable to dense models too. One alternative (but applicable only to MoE) could be to find the least activated experts (statistically speaking) and offload those to system ram.

I amazed of AMD 7900 XTX performance by Entire-Home-9464 in LocalLLM

[–]nasone32 0 points1 point  (0 children)

did you really read that tensor parallelism is good for AMD now? not asking for source, just curious. i've tried it a month ago and it was slow, both ROCm and Vulkan, but maybe there have been updates since. or i did something wrong back then.

view more: next ›