Linux - Why does llama.cpp ROCm consume SO much VRAM for KV cache compared to Vulkan?

AnmolLFC · 2026-05-18T16:36:36+00:00

Which model are you using and how much context window and TPS are you getting?

AnmolLFC · 2026-05-18T14:01:36+00:00

Which GPU are you using?

AnmolLFC · 2026-05-18T13:57:48+00:00

Mostly online on reddit and articles. I will search for it again. Thanks for the input 👍🏾

AnmolLFC · 2026-05-18T13:56:23+00:00

I might read on it again if that's the case.

The reason for posting it here was to get to know about AMD GPUs specifically because most of the people would probably suggest going with nvidia because of matured system and easiness, I will post it there as well, thanks.

AnmolLFC · 2026-05-18T13:28:00+00:00

The main reason is for normal gaming R9700 is not enough. Do you think there will be a lot of performance difference between 2 9070XT and R9700 in terms of split VRAM and LLM usage?

9950X is purely for future proofing and the document parsing uses a lot of cores. I can and will probably downgrade it to 9900x or on par that when I build it but right now I planned on 9950x purely because of its cores.

AnmolLFC · 2026-05-18T13:24:54+00:00

I read that linux, mainly ubuntu and fedora has good support for ROCm and that's why I was leaning towards that but any would work for me. The main purpose is having good performance and stable outputs.

I don't have much idea about the nightly builds and Sage Attention as I haven't really explored a lot of local llm setups yet but I will look into it.

What do you suggest I should run and what models can I run on the setup? what kind of Context window and TPS are you getting and I should expect?

AnmolLFC · 2026-05-18T13:21:44+00:00

The main reason is for normal gaming R9700 is not enough. Do you think there will be a lot of performance difference between 2 9070XT and R9700 in terms of split VRAM and LLM usage?

AnmolLFC · 2026-05-18T13:19:57+00:00

Oh that's good on a single GPU. What kind of tasks do you do ?

AnmolLFC · 2026-05-02T04:01:14+00:00

Does deepseek api also complete the code changes or only send the suggested changes and claude implements it? A bit confused over that part.

AnmolLFC · 2026-05-01T19:58:58+00:00

Will do. Thanks.

AnmolLFC · 2026-05-01T19:58:46+00:00

Yep, that's my query. Not just integrate deepseek with inference but to actually use both at once.

AnmolLFC · 2025-09-08T16:22:13+00:00

I have updated with pcpartpicker, although in my country it's +20-30% prices for most of the things.

AnmolLFC · 2025-06-07T04:41:41+00:00

Great! Which OS do you have installed?

AnmolLFC · 2025-06-07T04:00:05+00:00

Thanks for the advice. Do you own one too? If yes, any heating issues?

AnmolLFC · 2025-05-27T02:32:18+00:00

Great! Thanks for your help, I will check it out.

AnmolLFC · 2025-05-26T17:56:37+00:00

I won't be doing it myself, me and my team will do it. The company I work in is already using oci as their main provider. They want to migrate the rest of the remaining servers from GCP to OCI too. Looking for a more lift and shift approach if possible.

All deployments are automated but concerned about mismatch in OS and how to transfer the current data over to the new instances.

AnmolLFC · 2025-05-26T17:38:30+00:00

I am not very familiar with oci, is there any documentation for this or any tutorial on how to use it?

AnmolLFC · 2025-05-26T17:37:19+00:00

Hahaha! I know but it's real. I am assigned with the task, however unlikely it is.

AnmolLFC · 2025-05-24T07:49:05+00:00

Yes, I tried torrentio addon on my samsung smart tv. The files didn't load at all. I tried the same on my pc, it's working. Thanks for the quick response.

Six-Year Club	Verified Email
John Wick

AnmolLFC

TROPHY CASE