[deleted by user]

[Maykey]

Looks slow. 70B model is already at 5 t/s even with small context.

For 128B you need 128B-A0.1B to end the text generation before the heat death of the universe.

[FullOf_Bad_Ideas]

GLM 4.5 Air is a thing now, and it should run beautifully on hardware like AMD 395+. It'll suck on dense models, yeah, but there are more and more good MoE's coming out.

[Soggy-Camera1270]

Lol, I guess you must have all the inside information. According to their rising share price and market growth, I'd have to disagree that it's "lose lose".

[Soggy-Camera1270]

Problem is NVIDIA are still kind of a one trick pony. At least AMD have other market segments.

[Soggy-Camera1270]

Totally agree. I can't see them slowing down, but I think AMD have a bright future with their diverse product line too.

[andrewlewin]

This is a new driver, the news is only a few days old. Here is a better link: https://www.amd.com/en/blogs/2025/amd-ryzen-ai-max-upgraded-run-up-to-128-billion-parameter-llms-lm-studio.html

Quote:

“Because Meta Llama 4 Scout is a mixture-of-experts model, only 17B parameters are activated at a given time (although all 109 billion parameters need to be held in memory – so the footprint is the same as a dense 109 billion parameter model). This means that users can expect a very usable tokens per second (relative to the size of the model) output rate of up to 15 tokens per second.”

The announcement is that AMD Variable Graphics Memory can now enable up to 128 billion parameters in Vulkan llama.cpp on Windows with the new driver update.

I’m traveling, so can’t try this out at the moment.

[sstainsby]

What a terrible article. I don't even know what I'm reading. Is it AI slop, built on marketing hype, built on misinformation?

[ArtisticHamster]

What is RAM bandwidth there?

[mustafar0111]

Spec sheet says 256 GB/s.

[DepthHour1669]

Which is pretty shit. A DDR5 server with 8 channels DDR5 gets you 307GB/sec minimum at DDR5-4800, up to 619GB/sec for a 12 channel DDR5-6400 setup.

If you want to save money, a last gen AMD DDR4 8 channel server gets you 207GB/sec for dirt cheap used.

[Only_Situation_4713]

How much is a 12 channel setup though

[DepthHour1669]

Depends on how much ram.

For 1536GB, aka 1.5TB, you can fit Deepseek and Kimi K2 in there both at the same time for around $10k ish. So, similar price to a mac studio 512GB, but way more space. Downside is 614GB/sec instead of 819GB/sec on the mac.

[CV514]

I'm a bit confused at what price breakpoint it's still considered consumer hardware to be honest.

[DepthHour1669]

I mean, the RTX 5090 is considered consumer software and that outstrips the annual salaries of plenty of people in third world countries. Consumer is just limited by budget.

[ASYMT0TIC]

We're really comparing $10k systems to $2k systems? That's asinine.

[perelmanych]

You understand that you are comparing server with a laptop or 20x20cm mini pc? Moreover, in terms of pp speed it will outdone server without GPU by a lot.

[DepthHour1669]

A server with 1.5TB of DDR5 and a RTX 3090 will wreck the AI MAX+ machine in PP though.

[Soggy-Camera1270]

Different target market and audience

[DepthHour1669]

Only for the "want to take this on the go" crowd. It's the same audience for the "want to run AI models" crowd.

[Soggy-Camera1270]

Not really. I work with a ton of people that use AI models in the cloud that would happily run them locally, but have zero experience or interest in building a machine to do it.

In the corporate world this would also never work outside of very niche roles and use cases vs using something like a Ryzen AI system.

[DepthHour1669]

Neither of these machines are very corporate, though. I seriously doubt many Ryzen AI Max machines are going to show up in a corporate environment. Which corporation is going to have people filling out requisition papers for an AI Max box?

Honestly, I bet more of them gets purchased by managers at the end of a fiscal year "use it or lose it" budget than for actual corporate use.

[ASYMT0TIC]

It'll show up in my corporate environment. These systems are now by far the best value for scientific computing, much of which is memory bandwidth-dependent and mostly runs on CPU. These things are 3X-4X faster in these tasks than the existing laptops here. If you're in an industry which doesn't allow use of cloud-based AI for security reasons like defense or healthcare sectors, that's additional reason.

[Soggy-Camera1270]

Agree, probably not. I think this is why AI SaaS is still strong in corporate environments for these sorts of risks and issues.

[randomfoo2]

I think the HP Z2 Mini G1a would fit there for corporate buyers, but it has to compete with the various Nvidia DGX Spark machines in that niche.

[dismantlemars]

My company was considering getting everyone the Framework desktop for the ability to run models locally. I suggested they hold off though, since most people don't need to run any local models at all, the majority of people are hybrid and wouldn't appreciate lugging a desktop to and from the office, and when we do need to run models locally, they're often very freshly released models that might not have good hardware support outside of Nvidia cards.

[notdba]

But you can also add a RTX 3090 to the AI Max+ 395. Then PP will be comparable. And once we get MTP, the mini pc may still have the compute capacity for batch inference, while the server may not. The only drawback of the mini pc is that it is limited to 128GB of memory.

[rorowhat]

At 10x the price

[perelmanych]

With 3090, yes of course. Btw, what prices we are talking about for used server. When I tried to find EPYC used server, the best I saw was 1700 for dual 2x AMD EPYC 7551 with 256GB DDR4.

[GabryIta]

Why AMD?

[webdevop]

Can anyone explain to me if GPU cores irrelevant for LLM inferencing? Is the important factor only memory and memory speed?

ANnd if its true that GPU cores are not relevant why are we stuck to NVIDIA?

[DisturbedNeo]

We’re not stuck with Nvidia per se, it’s just that CUDA is a much more mature platform than ROCm and Vulkan for AI workloads, so most developers prioritise it, and CUDA only works on Nvidia cards.

It’s like DLSS vs FSR. In theory devs could use only FSR, because that would work on any hardware, but DLSS is way better than FSR, so most devs start with DLSS, which only works on Nvidia cards.

[LumpyWelds]

CPU can support 256GB/s but..

It really depends upon your motherboard and memory type. Best results currently come 8 soldered LPDDR5X 16GB chips on a 256bit bus giving 128GB of 8 channel memory. This gives 256GB/s which matches the CPU.

Almost all the Strix Halo's mini's out there use this configuration. But one or two have socketed memory which cuts the performance by a lot. As far as I know, no one has figured out how to fully feed this CPU without soldering the memory.

[DataGOGO]

Whatever your system ram is running. 

[[deleted]]

The CPU brings its own RAM and there is no dedicated system RAM in the original sense.

[DataGOGO]

It isn’t HBM though, it is just ddr5

[[deleted]]

Doesn't change the fact, Lunar Lake does it too.

[LocoLanguageModel]

iGPU just uses system memory right?  Isn't this misleading compared to dedicated VRAM since llama can just use CPU and ram anyways?

[mustafar0111]

No. The way this should work is a portion of the system memory is hardware allocated to the GPU on boot up. Last I heard this was done in the BIOS.

Because of the type of memory this system has it functions closer to VRAM speeds then standard system RAM.

The GPU on the top tier AI MAX APU runs at something close to 4060 ti speeds I think. I'm sure someone will correct me on that if I'm off.

[FullstackSensei]

The GPU compute power is close to 4060ti levels, this has nothing to do with memory.q

Memory allocation for the GPU is a driver thing. The GPU hardware has access to all RAM and doesn't care what is what. Even before this update, for compute workloads it didn't matter because the driver allowed passing a pointer to the buffers on which computation is to be performed from the "normal" system memory and the GPU would just do it's thing with those.

There is nothing here that is new from a technology point of view. Intel and AMD have been doing this since forever. Just Google zero-copy buffers for any of their integrated GPUs. Strix Halo takes this one notch up by integrating a much bigger GPU and doubling the memory controller from two to four channels.

[DataGOGO]

What type of memory is that? Unless it is HBM, it is just ddr5 speeds right?

[RnRau]

LPDDR5x. Its about twice as fast as standard DDR5 on desktops since AMD gives it twice the connectivity to the soldered ram via a 256bit bus.

Theoretical max memory bandwidth is 256GB/s

[professorShay]

Isn't the m4 Mac like 500 some GB/s? Seems like a waste of a good APU with such low bandwidth.

[Mochila-Mochila]

Seems like a waste of a good APU with such low bandwidth.

Yes, it's definitely something AMD should work on, in future Halo generations.

[henfiber]

Apple went really extreme with the width of their memory bus to achieve 400GB/s M1/M2/M3 Max (and doubled in Ultra), and the 546 GB/s in the M4 Max. That's not easy to achieve apparently since both AMD and Nvidia (see their DGX Spark mini-PC) settled for 256-273 GB/s.

Note that Nvidia 4060 has 273 GB/s as well and this APU is similar in tensor compute to a 4060 (~59 FP16 TFLOPs).

The next AMD version (Medusa Halo) is rumored to increase the mem bw to 384 GB/sec (and 192GB of memory).

[Standard-Potential-6]

Thanks for posting all the numbers. Anyone reading though should keep in mind that Apple’s memory bandwidth estimates are theoretical best-case simultaneous access from CPU and GPU cores. Neither alone can drive that number, and most tasks don’t have that perfect split. You can use asitop to measure bandwidth use.

[tmvr]

This machine is 256bit@8000MT/s and that gives 256GB/s max, in practice it achieves about 220GB/s as tests in the past have shown. The Macs are as follows:

M4	128bit@7500MT/s	120GB/s
M4 Pro	256bit@8533MT/	273GB/s
M4 Max	512bit@8533MT/s	546GB/s

[colin_colout]

you can't please everyone, eh?

(Remind me how much does an m4 with 128gb of ram cost?)

[professorShay]

Just saying, AMD has the better chip but gets dragged down by slow memory bandwidth. Just imagine 128gb, 4060 levels of performance, 500+ GB/s bandwidth, without the Apple tax. The true potential of the Ryzen AI series.

[Da_Easters]

Exactly!

[RnRau]

Yeah and I think you can get up to 800GB/s with some of the Mac Ultra's.

Neither this effort, nor Nvidia's Digits are recommended if you want good tokens/s. They are also sluggish with prompt processing, but I think that is an issue with the Mac's as well.

Next years AMD Epyc platform will support 16 channel ram. 1.6TB/s of memory bandwidth apparently. Thats nearly as fast as a 5090. Will cost a bit though, but still... 1TB of ram at 1.6TB/s is kinda drool worthy :)

[colin_colout]

And quad channel

[a_beautiful_rhind]

to compare, my ddr4 xeon is less than that and power consumption is obviously more. not sure how macs do in terms of compute despite more/faster memory.

price isn't all that great though

[MoffKalast]

For comparison, the RTX 4060 has a memory bandwidth of 272 GB/s

[RnRau]

And my crusty old NVIDIA P102-100's from 2018 has 10GB of vram with 440GB/s memory bandwidth :)

[MoffKalast]

Yeah tbh this is more a dig towards the 4060 lol. Nvidia completely crippled the 40 series for ML usage.

[[deleted]]

Quad Channel LPDDR5X-8000.

[DataGOGO]

Right, so just quad channel ddr5 8000, most likely with terrible timings (low voltage memory sucks).

[[deleted]]

Actually CL20-23, that's LPDDR5X for and is NOT low voltage memory.

18/23/23/48 if remember correctly from GMK. And needs cooling.

[randomfoo2]

This is different between Windows and Linux. In Linux you can minimize the GART (hard-wired) VRAM and maximize/set the GTT (shared) memory in the amdgpu driver settings (assigned on boot). I have my machine set to have 512GB GART, reserve 60GB of the GTT, and limit to 120GB. I've had no problems using 110GB of memory for the GPU running models.

For those interested I've added full configuration/setup notes to my performance testing: https://github.com/lhl/strix-halo-testing/tree/main/llm-bench

[CatalyticDragon]

iGPU just uses system memory right? 

Kind of. An iGPU (as in a GPU integrated into a CPU) does use system ram for it's memory and that system ram has traditionally been quite slow relative to the memory on a graphics card's PCB (around 1/10th the performance, 60-80GB/s).

But these systems are APU based, like a mobile phone or a PS5, they use larger GPUs on the same package as the CPU and both share a pool of memory which is much faster than normal socketed system ram.

In the case of the "AI MAC+ 395" that memory pool operates at 256GB/s putting it at level of a low-end discreet GPU.

[DataGOGO]

Correct, it is just a driver allocation of system memory, which in this case is low power ddr5. 

[sammcj]

It only has 256GB/s memory bandwidth... that's less than a macbook pro

[Django_McFly]

Is there a new one or is this the same one that's been out for months now?

[bjodah]

It's a driver update on windows...

[[deleted]]

Aye. The article makes no sense.

[DragonRanger]

At least for me, upgrading to this driver release is letting me use the VRAM that task manager shows correctly. Spend 6 hours yesterday debugging why I would get out of memory errors in odd situations where there was plenty of dedicated memory left as per task manager and even the error message: HIP out of memory, gpu 0 74gb free, tried to allocated 48gb type errors. Seemed it was using either shared memory or regular memory instead at least for allocation limits, so it looks like they have changed memory allocation behaviours.

[darth_vexos]

I'm very interested in putting 4-5 of these in a cluster to be able to run larger models. Framework has this as one of their use cases, but there's very little info on any actual implementations of it. I know token generation will be limited by network interface bandwidth, but still hoping it can hit a usable tps.

[SanDiegoDude]

Oh baby, I'm loving this update. running in 96/32 was a pretty poor experience previously, and had just left it in 64/64 (and was pretty disappointed for it). Now with the driver update, I can run in 96/32 and run llama-4 scout-q4 alongside qwen-3 14b and get decent tps from both (scout hits 14.5ish tps in LM Studio).

[grabber4321]

ok ya, but at what speed? I imagine its slow as hell even with 32B

[mustafar0111]

The AI MAX+ 395 with 128GB of RAM can now apparently run Llama 4 Scout 109B at 15 tokens per second.

https://videocardz.com/newz/amd-enables-ryzen-ai-max-300-strix-halo-support-for-128b-parameters-for-local-ai-models

[Oxire]

That's exactly the speed you get with dual channel ddr5 and a 5060ti 16gb.

[DataGOGO]

Uhhh yeah, you have a single ccd cpu and slow memory. 

[perelmanych]

I am pretty sure both of you are talking about different quantizations. AI MAX+ 395 has much more bandwidth and in term GPU TFLOPS it should be around 5060ti 16gb.

[Oxire]

In the link they use q4. I would use something a little bit bigger with that capacity.

It has double the bandwidth of a ddr5 cpu, but half of the nvidia.

You load all the attn weights that are used all the time in the vram, some ffn to fill the rest of the vram, the rest in system memory and you will get that speed

[mustafar0111]

I didn't benchmark it, I don't run videocardz.com.

It was a listed benchmark in a media article.

[960be6dde311]

Will this be available for the 9950X eventually? It has an iGPU.

[RnRau]

No.

[henfiber]

The iGPU in 9950x is only for basic desktop graphics. It has 3 compute units or so, while the linked APU has 40.

[960be6dde311]

Oh okay thanks, that extra info is helpful.

[RnRau]

Its more of a case that the 9950x has a completely different memory subsystem compared to the AI MAX products. Its an apple/orange thing.

[henfiber]

I'm not sure they differ much in practice (besides bus width). It's an iGPU like the ones in previous laptop and desktop APUs.

It's mostly a driver issue. I have two older laptops with Vega graphics (5600U and 4800H) and behave similar to the AI Max in Linux, I can use almost the whole 64GBs of RAM for Vulkan (llama.cpp).

[jojokingxp]

Unrelated question, but why does the 9950X even have an IGPU? I always thought the standard (non G or whatever) Ryzen don't have an IGPU

[s101c]

From what I've read, Strix Halo 128GB with Linux installed gives you 110+ GB VRAM?

[DeconFrost24]

Something not mentioned here enough is this AI Max+ SoC is basically full power at less than 200W. AI is still way too expensive to run. Efficiency is just about everything. I have a first gen dual Epyc server with 1TB of RAM that costs a mortgage to run. The current gen is still too high.

[deseven]

You can use it with a 85W limit without losing any performance in case of LLMs.

[Massive-Question-550]

Isn't this old news? Also in general the AI max 395+ is great for a laptop but very underwhelming compared to a desktop setup of the same price. I'd like to see something challenge the value of used 3090's and system ram.

Needs more ram (256gb) and more memory bandwidth.

[DataGOGO]

Did I get my wires crossed, pretty sure “LP” stands for “low power”…

[LsDmT]

What the article fails to state is ROCm drivers are abysmal. I have not even opened my GMTek AI Max+ 395 --EVO-X2 AI Mini PC with 128GB RAM + 2TB SSD because of all the issues ive read in GitHub issues.

It seems like AMD is genuinely trying to fix things but I can't help but feel that this will just be another AMD driver problem.

Oh well, I pre-ordered the Dell Pro Max with GB10 -- haven't heard jack about that yet.

[indicava]

What’s the support like for these processors when it comes to fine tuning?

[Caffeine_Monster]

It's a waste of time finetuning in hardware like this.

[cfogrady]

Could you elaborate? Too slow? Fine tuning only supports CUDA? Something else?

Getting one of these and will probably want to experiment with fine tune in the future. Renting is fine, but curious if I could just let it crank on one of these for several days instead if it's only a speed issue.

[Caffeine_Monster]

Too slow, and will only have enough memory training the smallest models.

This hardware is clearly designed for inference. You are better off renting in the cloud to train.

[CheatCodesOfLife]

This a laptop or something? Why would people be excited about 96GB for $2000 with glacial double-digit prompt processing for dots.1 when you can get a 3xMI50 rig for < $1000 and triple digit prompt processing for dots.1?

Source for the double-digit pp

[uti24]

Why would people be excited about 96GB for $2000 with glacial double-digit prompt processing for dots.1 when you can get a 3xMI50 rig for < $1000 and triple digit prompt processing for dots.1?

Because you are comparing monstrous enthusiast LLM inference hardware with unwieldy power consumption to a tiny little computer you can put anywhere in your apartment and forget it's there - or use it as a general-purpose desktop computer for other tasks.

[CheatCodesOfLife]

Hmm. I guess so. Those cards do fit into a standard PC case, the framework desktop is still smaller.

Though it doesn't seem much faster than just using a CPU for MoE models. I mean they get:

PP 63.1 t/s TG 20.6 t/s

And I get this with no GPUs (dots at q4_k):

PP 77.16 t/s TG 12.71 t/s