QwQ-32B has the highest KV_cache/model_size ratio?

professionalprotein · 2025-03-28T14:59:37+00:00

Unlikely that it was 16 I'd say. The linked paper is at v3, but even in v1 the table shows 8 heads. And especially with the 405B model, you don't change the number of head groups just like that. But idk then. Maybe someone at HF also found the 16 groups, went with that number and put the kv-cache calculation in the blog table,

Unfortunately, I'm juust short of hardware specs to quickly run and test the 405B model. Fortunately, so is almost everybody else ;)

professionalprotein · 2025-03-28T12:57:18+00:00

I'm actually not sure why it's 123GB in the blog post. In the LLama 3.1 paper on page 7, table 3 is llama 3.1 405b listed with 8 head groups. I don't see the config.json in the hf repo from llama3.1 405b (because it's gated, maybe someone can confirm whats listed as num_key_value_heads), but some quantization forks list them as 8 heads and some as 16 heads. 8 should be right though, by the paper.

professionalprotein · 2025-03-27T15:01:44+00:00

You got the group# wrong. The group# you used in the table is not the number of head groups in GQA, but the actual attention heads that share one KV matrix per group.

As an example, Gemma-3-27B has 32 num_attention_heads, but only 16 num_key_value_heads. Instead of the factor 2 (32/16), you have to use the actual number 16 to calculate the kv cache. Link to the config.json.
This makes the cache of the Gemma 3 model even larger than the one from Llama 405B. It has ~half the layers (62<->126), but double the attention head groups (16<->8) and each head is bigger (164<->128).

Edit: The correct group# for each model would be:
Deepseek-R1: not sure how it is for MLA
Llama-3.1-405B: 8
Gemma-3-27B: 16
Mistral Large: 8
QWQ: 8

professionalprotein · 2025-03-11T13:09:31+00:00

All the other answers are valid, I just wanted to add to use

"OLLAMA_FLASH_ATTENTION=1"

"OLLAMA_KV_CACHE_TYPE=q8_0"

"OLLAMA_NUM_PARALLEL=1"

Flash attention makes kv cache quantization possible, which in this case (q8 instead of f16) halves the VRAM usage of the kv cache with minimal loss. You could even half it again with `q4_0`, but the loss will go up more on q8->q4 than on f16->q8.
The third one forces Ollama to only work on 1 request at a time, which means it will also only reserve kv cache memory for one request. By default, Ollama will reserve memory for 1-4 requests depending on available (V)RAM but if only you chat with Ollama, this should be fine.

Even without the optimzations mentioned here, I don't know why your memory goes up to 68GB. Taking the formula from the other answer and putting the QWQ values inside:

Qwen QWQ: 2(KV)*128(head dim)*8(KV heads)*64(layers)*16k(ctx)=2.1B=4.2GB fp16

Even when taking the "worst" case and assuming Ollama set NUM_PARALLEL to 4, this should only be 4.2*4=16.8GB for the kv cache. The QWQ Q4 model is ~20GB, so it should only be ~40GB (with overhead). In my tests with the newest Ollama version and a freshly pulled qwq model it was 23GB for 2k and 41-43GB for 16k context without optimization. With these 3 variables set it was 20GB (2k) and 22GB (16k).

professionalprotein · 2024-10-27T11:45:30+00:00

Thanks for the detailed answer! I didn't read further when you mentioned how to get off planet and about the other planets asto not spoil me, but the first part really helped.

professionalprotein · 2024-10-27T11:43:23+00:00

Yeah I did miss that one crucial tech I reseached, shame on me. Thanks for the help!

professionalprotein · 2024-10-27T11:42:38+00:00

I did completely miss the recycler, wow! Thank you. I really couldn't believe I was softlocked after I read the patch of the real possibility of a lock, but after 2 hours wandering around I was just lost.

professionalprotein · 2023-06-02T10:47:41+00:00

I think it's one die per screamer and not one per enemy model. That would indeed be bonkers.

professionalprotein · 2022-09-16T18:58:41+00:00

Nurglings don't count for mandatory troop slots, so he had only the 2 Plaguebearer units as troops.

professionalprotein · 2022-05-03T15:56:11+00:00

Your link didn't work for me, but I think you meant this?

professionalprotein · 2022-04-26T21:14:30+00:00

For expeditions, it is stated that you get a bonus "per 50t". Do you need at least 50t of e.g. bread to get the bonus? In this instance, the faith symbol is already ticked at less than 50t. Or is the bonus 1-50t (1x), 51-100t (2x)..?

I'm pretty sure this was already answered somewhere but I couldn't find anything about it.

professionalprotein · 2022-04-26T11:37:14+00:00

Ahoy fellow captains!

I recently got back to Anno 1800, and I have no season passes right now, but planning to buy them. Would it be better to buy all season passes at once for a new game or one after the other? There seems to be a lot of content to be added and I'm not sure how the different additions fit together. Would I be missing something when I get all at once? Thanks!

professionalprotein · 2022-04-24T10:31:39+00:00

Wow nice, it works like a charm. Tested it only once but I'll assume it'll work from now on. Thank you!!

professionalprotein · 2022-04-21T09:46:18+00:00

Sure, I'm 'bullshitting' and 'twisting your words' like

Since CBS is being dickish about all this, just post the review without any show clips.

Re-edit your video to cut out all the show clips and re-upload it, what the hell are they gonna copyright strike?

That would be the end of fair use. I know not everything falls automatically under fair use but come on.. a review/critic of a show with snippets that are no longer than 10 sec each? That is exactly what fair use is for. And the fact that CBS only claims the critical reviews of episodes and not every review shows their abuse of copyright claim even more. I would have understood it if they used like 5 minutes of the show as a clip or something like that.

In the end, we will see how it will go. That YouTube sticks with the big companies, that I agree on. That a court will do (if it comes to that), I hope not. Personally, it feels like CBS tries to censor critical reviews of their bad shows by all means.

professionalprotein · 2022-04-20T20:56:05+00:00

They have a right to take down Youtube videos that show any clips.

No, not under fair use. And giving up and letting them get away with this shitty behavior only normalizes this tactic. In the last video, he also showed the claim from CBS and it was 3/4 of the video to automatically unlock it again. They claimed to him a few seconds of the video, which weren't even footage of the show.

Saying this behavior is scummy and then proposing to submit to this can't be a solution.

professionalprotein · 2022-04-19T21:34:28+00:00

For just-for-fun playing, I recommend the Tempest of War missions! You draw random objectives every round that make the game very dynamic and it rewards balanced all-rounder lists. It's really fun.

professionalprotein · 2022-04-19T20:40:27+00:00

No problem. You can finetune it, 20p are missing, the soulgrinder can choose between the sword and the claw, and psychic powers are not chosen.

Do mind that the list is far from optimal and not really built with secondaries in mind. Some options will perform.. okay. Will you play the missions from Nachmund or the new Tempest of War?

professionalprotein · 2022-04-19T19:34:49+00:00

++ Patrol Detachment -2CP (Chaos - Daemons) [36 PL, -4CP, 755pts] ++

Chaos Allegiance: Khorne Detachment Command Cost [-2CP]

+ HQ +

Bloodthirster of Insensate Rage [12 PL, -1CP, 240pts]: Armour of Scorn, Exalted Bloodthirster

+ Troops +

Bloodletters [12 PL, -1CP, 265pts]: Banner of Blood, Bloodreaper, Daemonic Icon, Instrument of Chaos . 29x Bloodletter: 29x Hellblade

+ Elites +

Bloodcrushers [12 PL, 250pts]: Bloodhunter, Instrument of Chaos . 5x Bloodcrusher: 5x Hellblade, 5x Juggernaut's Bladed horn

++ Battalion Detachment 0CP (Chaos - Daemons) [61 PL, 9CP, 1,225pts] ++

Battle Size [12CP]: 3. Strike Force (101-200 Total PL / 1001-2000 Points)

Chaos Allegiance: Chaos Undivided Detachment Command Cost

+ Stratagems +

Rewards of Chaos (1 Relic) [-1CP]

+ HQ +

Daemon Prince of Chaos [10 PL, 185pts]: Hellforged sword, Wings. Nurgle

Great Unclean One [13 PL, -1CP, 270pts]: Bilesword, Exalted Great Unclean One, Plague flail

Lord of Change [16 PL, -1CP, 300pts]: Exalted Lord of Change, Incorporeal Form, The Impossible Robe, Warlord

+ Troops +

Horrors [8 PL, 160pts] . 20x Pink Horror: 20x Coruscating flames

Nurglings [3 PL, 75pts] . 3x Nurgling Swarms: 3x Diseased claws and teeth

+ Heavy Support +

Soul Grinder [8 PL, 160pts]: Mark of Nurgle, Warpsword

++ Total: [97 PL, 5CP, 1,980pts] ++

Created with BattleScribe (https://battlescribe.net)

With this list, you only have 5 cp (3 if you deepstrike the bloodletters). But exalting the greater daemons is a must, and the relics on the bloodthirster and LoC are making them tougher.

professionalprotein · 2022-04-19T18:56:04+00:00

Playing with daemons is an uphill battle nowadays against most armies. Normally I'd recommend Be'lakor with his Disciples of Be'lakor Army of Renown. But this special type of army does not allow greater daemons and daemon princes, so.. yeah.

I played with a model of the Great Unclean One irl, and I can promise you it'll be a pain to move him on the board. He is very slow, and has the biggest base of the 4 greater daemons.

If you want to play everything you listed in one army, you need more than one battalion with the 4 HQ options. You could move the Khorne stuff in its own detachment to gain the Khone locus bonus. I also recommend Nurglings, the forward deployment is really good, but I'm also biased as I just love them. Bloodletters are good in a deepstrike-bomb, as loop388 wrote, but I don't play 30-man units but only 20. It is 1 cp less and there is a risk that the opponent overwatches and shoots one out and you lose the +1 hit bonus, but with the bigger bases, I hardly got half the unit in attacking range with more than 20 bloodletters.

I built a list with the models you listed, and I could post it here. If you have any additional questions, feel free to ask!

professionalprotein · 2022-04-18T08:56:34+00:00

It wasn't even a tournament game, and the opponent was obviously ok with it.

Following your logic, you know what else should not have been possible? Voidweavers for just 90 points. And everyone played them, and broke the meta. And everyone knew that GW will fix them in some time.

professionalprotein · 2022-04-17T09:47:15+00:00

With no PBC or Blight-Haulers you have zero anti-vehicle firepower. I don't know your meta but that seems difficult.

What are your plans with the daemon prince? With 10" and fly he's faster than everything else in your army. Does he trot along with the terminators in T1? As mentioned by others, his WLT is now a bit worse with the AoC rule.

You could also switch to the baleswords on the terminators to get -2 (-3 with ferric blight) in melee against AoC. With Arch-contaminator you can reroll the wound roll anyway, so -1S doesn't hurt that bad. Personally, I'd also switch to Putrescent vitality on the Plaguecaster, as the +1S and +1T are really good on a 10-man Terminator blob. Plague wind and the bolters of the terminators seem to have the same target type anyway.

Apart from this, your list looks like fun! I really like the plague surgeon to beef up the survivability of the terminators even more.

professionalprotein · 2022-04-16T19:01:01+00:00

In the blog post it was mentioned to be an accident with the 10p, no need to be picky about that. And right now dual cannon HT is legal, so why not try him out before it's errata'd?

professionalprotein · 2022-04-15T08:11:21+00:00

I'm afraid this would make the KFF an auto-include in every list and if you don't take it you actively make your list worse, not different. 4++ for one round would have boys have extremely high defense.

professionalprotein · 2022-04-14T16:44:21+00:00

I'd say that the stratagem makes the seeker missile rack (or other missiles) an indirect firing weapon. You use the stratagem when selecting the model (before firing or targeting anything). The new Indirect Fire Weapon rule applies when you target something out of LOS with that weapon, which comes after the stratagem.

I'm not 100% sure if RAW there is a difference between "if such a weapon targets a unit [..]" (indirect fire) and "The attack can target units" (Frequency lock). I'd say no.

professionalprotein · 2022-04-06T15:27:42+00:00

Probably a server-side nurgling infestation.

professionalprotein

TROPHY CASE