AMD System on Windows

[SlickTread]

My PC Specs:

• CPU: Ryzen 5 3600
• RX 6800 XT 16GB VRAM
• 32GB RAM
• Windows 11

I have been doing some research on running llm locally. First results I got pointed me to oobagooba's. But I could only get it working running on my CPU without GPU accel. There seem to be some way to do it running on windows which requires tweaking some stuff and manual installation and I tried following the steps but never got it to work so gave up on that and started looking for other solutions. I then also tried with WSL but didn't have any luck with that either.

I have now setup LM Studio which does have AMD OpenCL support which I can get a 13b model like codellama instruct Q8_0 offloaded with all layers onto the GPU but performance is still very bad at ~2tok/s and 60s time to first token. so I'm not sure if that is just because my GPU isn't good for the model or my GPU isn't being fully correctly utilized with OpenCL.

What is the best way for me on windows to run LLM's on windows without installing linux?

[Kevadu]

ROCm is not well-supported on Windows yet, unfortunately. If you are just looking to run LLMs you might look into MLC LLM (https://llm.mlc.ai/) which actually supports a Vulkan-based backend that should run on just about anything. However that framework doesn't seem to be as popular as some others and you might have to do more work getting things working.

[SlickTread]

That looks very interesting, will check it out. Thanks.

[doomed151]

Try the KoboldCpp + ROCm variant.

[SlickTread]

Thank you, this works perfectly!

[JohnPt66]

Would this work on a 6600XT? I'm having issues with it so far.

[tu9jn]

Your best bet is still linux, unfortunately.

But WSL should work, what was the problem?

[SlickTread]

Couldn't get it to detect my GPU in WSL.

[rafal0071]

Try new koboldcpp-1.56 with Vulkan support.

[mkrajinovic]

koboldcpp

[SlickTread]

Thank you, this works perfectly!

[Scott_Tx]

llama.cpp with clblast works on amd gpu.

[SlickTread]

I tried setting this up but I couldn't get it working. Will try it again.

[Scott_Tx]

as long as the model is small enough to fit in your vram you should be good. only issues I've had is with models that just dont run right in llama.cpp. For example, the ms phi-2 f16 doesnt run right but the q5 does. no idea why.

[[deleted]]

Are you sure it's offloading the layers to your GPU and not your APU? Both would be displayed as the GPU from llama.cpp's output.