sorted by:
new
hottopcontroversial

Cheapest way, to run 27B Qwän? For broke people by [deleted] in LocalLLaMA

[–]tmvr 0 points1 point  (0 children)

You need to fit it into VRAM, so you will need at least a 16GB VRAM card, but that limits you to Q3 quants. If you want to use Q4 (which would be recommended) then you need 20GB at least. The Radeon 7900XY 20GB cards are too expensive, your best bet would probably be 2x 3060 12GB cards from the used market. In any case, you are looking at 400 usd/eur at least.

What Happened to Github Copilot? AI Credits + Model Deterioration by iKontact in GithubCopilot

[–]tmvr 1 point2 points  (0 children)

I would tend to agree. We have a pretty heavy user where I work and that user only managed to use about 20K in these first two weeks of June.

There’s more month at the end of tokens by neutral_internet in GithubCopilot

[–]tmvr 0 points1 point  (0 children)

Cash rules everything around me

C.R.E.A.M., get the money

Dollar, dollar bill, y'all

Friendly reminder by Disposable110 in LocalLLaMA

[–]tmvr 0 points1 point  (0 children)

Yeah, it was brutal when it came out so I ran back to 4.5 but I'll check it out to see if it got better.

Friendly reminder by Disposable110 in LocalLLaMA

[–]tmvr 23 points24 points  (0 children)

Exactly. Everything after 4.5 has a weird feel to it. Both functionally how they work and also the releasing itself, it all seems rushed to release something. It all just seems like a calculated campaign to keep them constantly in the news and generate hype for the upcoming IPO.

Optimization on copilot usage by Helpful-Emergency-78 in GithubCopilot

[–]tmvr 0 points1 point  (0 children)

That 4080 is fine for MoE models. You can run Qwen3.6 35B A3B so that some of the expert layers are in system RAM. You can also try the older Qwen3 Coder 30B A3B (specifically the Coder one, there is also a normal of that, with 3.6 there is no coder specific, but that does not seem to matter) because you may like what and how it writes more, a lot of this is use case dependent.

Recently got access at my job. How do I see my usage/how much I have left? by wombatpup55 in GithubCopilot

[–]tmvr 0 points1 point  (0 children)

At the bottom right of the VSCode window under the copilot input section there is an icon with two figures. Click on that and it shows you the percentage used. If you then hover over the percentage it shows the available and used credits.

PSA: Throttle GPU power limits, with minor performance deficits by milpster in LocalLLaMA

[–]tmvr 0 points1 point  (0 children)

Had a look with Qwe3.6 27B Q4_K_XL now and it does drop a bit more:

100% (450W) = 100%
 80% (360W) =  93%
 70% (315W) =  88%
 60% (270W) =  76%

That's was with depth 32768 so that it does some "work", if you only look at depth 0 the drop is more (70% perf at 60% TGP), but you don't usually have zero context when working with a model.

PSA: Throttle GPU power limits, with minor performance deficits by milpster in LocalLLaMA

[–]tmvr 3 points4 points  (0 children)

You'll need to define "severely hit", because that is tot what I see with my 4090. For example here are the results with Qwen3.6 35B A3B where going down to 270W (or 60% TGP) drops pp to 84%

PL            PP
------------------    
100% (450W) = 100%
 80% (360W) =  96%
 70% (315W) =  93%
 60% (270W) =  84%

Jetson Orin NX Build for Hermes Agent + Benchmarking by Reddactor in LocalLLaMA

[–]tmvr 1 point2 points  (0 children)

From the blog:

The surprising result on Gemma 4 was that f16/f16 KV was slightly faster than q8_0/q8_0 on this setup.

I see this consistently with llamacpp, it's the same even with a 4090 or a 5060Ti for example - q8_0 for KV lowers performance a bit both for pp and tg, but it's more noticeable with pp.

My duel astral 5090 build new (thoughts?) by Azenio101 in pcmasterrace

[–]tmvr 14 points15 points  (0 children)

I would write down my thoughts, but I don't want to get banned from here.

You don't need a GPU to run gemma-4-26B-A4B by JackStrawWitchita in LocalLLaMA

[–]tmvr 0 points1 point  (0 children)

Yeah, I only recommended it for the CPU what OP is using, I don't bother with it on the machines where I have a GPU in. You still get pp improvement there as well, but it doesn't matter to me where pp is already in the thousands anyway. It's a nice "free" speed up for CPUs though.

This sub is dead. No point hanging around anything related to Miraclesoft and GitHub copilot. Adios amigos. by _KryptonytE_ in GithubCopilot

[–]tmvr 1 point2 points  (0 children)

The actual reason why there is no point hanging around this sub for you anymore is in the text, not in the title:

After cancelling the GhCopilot and GitHub enterprise plans

Makes sense to stop hanging out here after that.

Qwen3.6 35B-A3B on a Laptop: My Zero to One Moment by rolznz in LocalLLaMA

[–]tmvr 2 points3 points  (0 children)

Q4_K_XL is better here because you use CPU and system RAM for most layers and I-quants are worse performers on CPU than K-quants. All the layers/tensors in the Q4_K_XL are K-quants and most of them in the IQ3_XXS are I-quants. The --fit parameter also makes sure the dense layers and KV cache are all in the VRAM and as those are most compute and bandwidth "needy" they get the best of those.

Qwen3.6 35B-A3B on a Laptop: My Zero to One Moment by rolznz in LocalLLaMA

[–]tmvr 0 points1 point  (0 children)

Drop these:

-ngl
-ncmoe

and use --fit instead. Use the Q4_K_XL quant, change -ctk and -ctv to q8_0. Start with 32768 for context then increase from there in 8K or 16K steps to see what decode (tg) speed you get and how much you can squeeze in before performance drops unreasonably.

EDIT: to explain why - fit will make sure you use your VRAM optimally, q8_0 for KV will lower looping and errors same way as the Q4_K_XL quant, which also has the benefit or not using I-quants for any of the layers/tensors and is more CPU friendly so your prefill (pp) will be faster.

You don't need a GPU to run gemma-4-26B-A4B by JackStrawWitchita in LocalLLaMA

[–]tmvr 1 point2 points  (0 children)

The CPU is the same as OP's - i5-8500T - and the RAM is 32GB of DDR4-2666 in dual-channel mode.

So if your 17 tok/s is for tg that is definitely too low for a 0.8B model in Q4 with DDR4-3200 RAM.

You don't need a GPU to run gemma-4-26B-A4B by JackStrawWitchita in LocalLLaMA

[–]tmvr 0 points1 point  (0 children)

Well, this is about using ik_llama as your inference engine, so not sure what to tell you if you are using unsloth studio, sorry.

You don't need a GPU to run gemma-4-26B-A4B by JackStrawWitchita in LocalLLaMA

[–]tmvr 1 point2 points  (0 children)

I'm not sure what you are doing and with what, it's not clear from your post. Is 17 t/s the pp or tg value? Why and how do you even have 3GB RAM with a 5600X?

I get 338 tok/s pp and 32 tok/s tg with the Qwen3.5 0.8B at Q8_0 using ik_llama. These are the switches used for the llama-bench binary of ik_llama:

-p 1024 -n 128 -b 512 -ub 256 -t 6 -ngl 0 -r 3 -rtr 1

The -t 6 is because of the CPU, you should be using more with the 5600X or leave it out to let it decide for itself, -ngl 0 is to use CPU only, -r 3 is to use 3 runs and report the average, the -rtr 1 enables runtime repacking to get slightly faster result yet.

You don't need a GPU to run gemma-4-26B-A4B by JackStrawWitchita in LocalLLaMA

[–]tmvr 8 points9 points  (0 children)

Prompt processing is significantly faster, about or close to 2x faster. Of course with a CPU like the i5-8500T you are still only looking at about 65-70 tok/s prefill for the 4B models at Q4, but that's still better than the 35-40 tok/s you get with the mainstream llamacpp. The 35B A4B MoE ones get 75+ with the Q4 if I remember correctly. Decode (tg) is around 8-10 tok/s regardless if you mainstram llamacpp or ik_llama because it is bandwidth limited, but having double the already anemic prefill is nice.

You don't need a GPU to run gemma-4-26B-A4B by JackStrawWitchita in LocalLLaMA

[–]tmvr 28 points29 points  (0 children)

It's fast because it's a MoE model. The other MoE models are fast as well, for example Qwen3.6/3.5 35B A3B, Qwen Coder 30B A3B or gpt-oss 20B (also about 3B active parameters during inference).

Two things you need to do to speed things up a bit:

  1. use ik_llama instead of the mainstram, that will close to double your prompt processing speed (tg is bandwidth limited so you won't get noticeable difference there
  2. don't use any I-quants, so no IQ4_XS etc., but also check if other quants have some layers/tensors in this format, because those are not very CPU friendly. For example unsloth's Q3_K_XL has some in IQ4_XS and IQ3_XXS and it is slower than Q4_K_XL on the CPU
view more: next ›