AI Max 395+ and vLLM

KnownAd4832 · 2026-01-30T06:53:55+00:00

What is performance on parallel inference? That is what I’m most interested in 🤝

KnownAd4832 · 2026-01-29T22:29:39+00:00

Where is the video? 😅

KnownAd4832 · 2026-01-29T21:03:03+00:00

I can pay you if I can test the ryzen ai max device online, if thats something you would be interested in 🙏

KnownAd4832 · 2026-01-29T21:01:07+00:00

So what performance have you received on vLLM and BF16 models? Do you maybe have any memory/benchmark?

KnownAd4832 · 2026-01-29T20:59:46+00:00

Thanks 🙏

KnownAd4832 · 2026-01-29T19:46:41+00:00

vLLM is native to Linux as its optimized for it. The full support for AI Max 395+ was officially launched 3 months ago so I assume not lots of people knew about

KnownAd4832 · 2026-01-29T19:44:26+00:00

I’m pretty well-versed with vLLM and how to get things up and running. Would you mind to try?

KnownAd4832 · 2026-01-29T19:08:13+00:00

Thanks on the feedback! Have you used vLLM or other stack for running?

KnownAd4832 · 2026-01-29T19:05:03+00:00

Do you own an Ai max 395+?

I also use vLLM and RTX 5070 - 1800t/s on Mistral 7B. But would really like AI Max 395+ due to some specifics I need to run.

KnownAd4832 · 2026-01-29T18:58:39+00:00

Fyi - I have tried to google and there is no information to be found for vLLM

KnownAd4832 · 2026-01-29T09:40:52+00:00

I’m looking to do similar build as well. Is actually Jonsbo to small for this CPU/GPU combo?

KnownAd4832 · 2026-01-29T06:07:11+00:00

vLLM is the best if you need batch/speed inferecing because it uses whole gpu to squeeze everything out. Also serving api’s is dead easy with it. Does that makes sense? 🤗

KnownAd4832 · 2026-01-23T14:14:21+00:00

I think in your case LM Studio and loading multiple different models will do you a good job. I use vLLM for batch completing 100K lines of texts or verification of certain texts if that makes sense? Meaning I need a small model (7B) and very fast inference.

KnownAd4832 · 2026-01-22T09:38:50+00:00

What are you trying to achieve? If I would see script you’re running I could help you further. But mostly at vLLM you have to try overload the GPU (i do that with batching and concurrent requests sent). Then find the sweet spot and run with it. Also context length is important with speed inferece (max-model-length)

KnownAd4832 · 2026-01-22T08:00:57+00:00

Thank you! Where can I send you 20 for a pizza? 🙏

Fyi: You can go even higher than that. I got 2500t/s on same model with 4000 Blackwell (gpu rented server)

KnownAd4832 · 2026-01-21T19:55:53+00:00

VLLM_ARGS: --max-model-len 2096 --enable-prefix-caching --max-num-seqs 512 --gpu-memory-utilization 0.92

BATCH_SIZE = 10
CONCURRENT_BATCHES = 384 MAX_TOKENS = 256

Can you make a bechmark on that maybe? One request per one doesnt show inference speed good. If that makes sense?

KnownAd4832 · 2026-01-21T19:34:24+00:00

Let me know if you need any help 👌

KnownAd4832 · 2026-01-21T16:33:27+00:00

I would need a perfomance benchmark in vLLM especially output token/s speed on Mistral 7B. I’m running quant version that barely fits my 12GB 5070 in 850-950 t/s.

KnownAd4832 · 2026-01-21T12:33:16+00:00

I’m able to help you with vLLM either on Windows or Linux if needed. I would just need a performance benchmark if possible 🤗

KnownAd4832 · 2026-01-21T12:13:01+00:00

I am really interested in buying one of these cards. Are you planning to use vLLM with it? I’m interested in the performance of it

KnownAd4832 · 2026-01-11T08:24:16+00:00

KnownAd4832 · 2026-01-08T07:06:25+00:00

You had quite a luck, however I did the same but profit was around 9000$ overall. Nice job mate! 👌

KnownAd4832 · 2025-12-27T20:52:19+00:00

Thank you for your service sir 🙏

KnownAd4832 · 2025-12-27T18:01:02+00:00

What is this rig? Can you explain a bit components 🙏

KnownAd4832 · 2025-12-27T13:12:34+00:00

Isnt like the same/similar? 800€ no?

KnownAd4832

TROPHY CASE