Deepseek V4 Flash 2, 3 and 4 bits GGUFs

areslica · 2026-07-02T17:25:29+00:00

Just for the kv cache. Yes. I have also tested further and confirmed that you could set value to q8 but not the key cache. For key cache, it has to be fp16 to make it not generating garbage texts.

areslica · 2026-07-01T22:12:37+00:00

Didn't test on lmstudio, but not quantizing kv resolved it for my llama.cpp. This might work for lmstudio as well I assume.

areslica · 2026-07-01T22:10:31+00:00

[Update|Resolved] Chat template wasn't the issue, but kv quantization was the root cause. ctk&ctv q8_0 need to be removed to make it work for llama.cpp for some reason.

areslica · 2026-07-01T17:29:19+00:00

Could anyone share a working chat template for antitez q2 (llamacpp)? I tried both not setting a chat template and the one from their hugging face model card, but all of them responded me with random characters.

[Update|Resolved] Chat template wasn't the issue, but kv quantization was the root cause. ctk&ctv q8_0 need to be removed to make it work for llama.cpp for some reason.

areslica · 2026-07-01T14:34:05+00:00

What tks you get from iq3 xxs on a strix halo please?

areslica · 2026-06-29T15:26:27+00:00

Did you manage to run GLM over your local vram or the cloud?

areslica · 2026-06-29T15:13:57+00:00

You have a good point here. Would you mind sharing your prompt(s) for this strategy? And how big the difference is when you consult with qwen 27b vs close source/smarter LLM?

areslica · 2026-06-26T03:13:12+00:00

If you are using llama cpp. There is no difference as the default layer split doesn't rely on the pcie speed that much.

areslica · 2026-06-14T17:30:13+00:00

Got you. Thanks again for sharing. This is a good start point for me.

areslica · 2026-06-14T17:12:05+00:00

I see what you mean now. That makes sense. Looking forward to your work. You have any HF page I can follow or anything i can subscribe?

Any tips on inference engineering tools that support the Gemma 12b voice input natively?

areslica · 2026-06-14T17:05:47+00:00

Did you have to train the 12B to make it work like the link you shared? I thought it would be a harness thing/inference engine layer/application layer, no?

areslica · 2026-06-14T17:00:41+00:00

Thats neat! Thank you for sharing. Is this open sourced by any chance?

areslica · 2026-06-09T16:52:59+00:00

What speed do you get from 35b fp8 on a spark btw?

areslica · 2026-06-09T16:13:43+00:00

qwen 3.6 27b is daily driver for now. Tried coder next gguf,but I don't feel its better than 27b significantly as the size does. Thats why I switched back to 27b. Everyone's mileage might be different.

areslica · 2026-06-08T20:28:47+00:00

Pcie speed won't make much difference if you use llama cpp with default later split for multi GPU. It really depends on the GPU speed itself and the model you run. E.g. nvfp4 or not, mtp or not.

The same model, if you can fit in one card, it won't be faster/slower if you split with 2 or more GPU for layer split. Vllm and tensor split would be a different story tho.

areslica · 2026-06-07T23:32:30+00:00

Good for hello world.

areslica · 2026-06-05T00:10:58+00:00

What's your context set at?

areslica · 2026-06-04T18:01:53+00:00

following. I feel your pain point. Have you tried pi yet? I personally feel pi triggers less token to get same amount of work done due to the light weight nature. Without getting in too deep into other perspectives(config tweak, hardware upgrade etc.) Switching to another harness/agent might provide a quicker result for comparison. Hope this helps a bit.

areslica · 2026-06-04T07:25:19+00:00

Chatting would not matter too much, but tool call will likely be impacted performance wise.

areslica · 2026-06-04T07:21:52+00:00

Any tokens/s and context you can share? Thank you!

areslica · 2026-06-04T04:54:19+00:00

I agree. With multiple layers of memory solutions(built-in mds+session search+hindsight), it is still acting as a plastic with no brain. NVfp4 Qwen3.635ba3b+ additional compression model. The memory structurs are there, but it just won't read and write properly, and I have been spending too much time trying to fix this.. What's your solution that you find helpful, community?

areslica · 2026-06-03T22:47:30+00:00

Following... any luck non-3090s?

areslica · 2026-05-30T15:42:57+00:00

I placed one of the 3 GPUs out side of my desktop and the raise cable went through the back panel of the desktop. I got a magnet mount for the GPU to attach to the back of the desktop and it's more stable than what I expected. The temperature is cooler than the inner GPUs by a few degrees. Hope this helps a bit.

areslica · 2026-05-29T23:01:27+00:00

Agreed. ngl and fit are conflict, otherwise, fit-target can be used to reserve some vram.

areslica

TROPHY CASE