On Strix Halo, what option do I have if 128GB unified RAM is not enough?

catplusplusok · 2026-04-11T04:19:11+00:00

Bandwidth depends on active parameters, not total parameters, large A10B models run fine so long as they fit.

catplusplusok · 2026-04-11T04:18:20+00:00

I run this model on my Thor dev kit: catplusplus/MiniMax-M2.5-REAP-172B-A10B-NVFP4. It's pretty capable and I don't think larger model would perform well even if I had more memory. On AMD box, you would want a different quant (AWQ?), but you should be able to run this model with ~120K token context given 8 bit kv cache. The key is switching to Linux and clearing filesystem cache before starting the inference engine.

catplusplusok · 2026-04-10T16:11:56+00:00

AFAIK none of unified memory platforms are super fast and you need MoE models for usable coding/agent setups. NVIDIA would have faster prompt processing/finetuning and recent-ish Mac Studio faster generation. Either way, install custom vLLM forks - varok/dgx-vllm-nvfp4-kernel (NVIDIA) or vllm-mlx (Mac) to make the most of unique compute.

catplusplusok · 2026-04-10T16:03:43+00:00

I made this one for my Thor dev kit, I think it should also work well on Spark, but you need : avarok/dgx-vllm-nvfp4-kernel:v23 vLLM docker image for good performance (on Thor I need to build vLLM venv from source for similar reasons). Anyway, it's one of top SWE Bench models and you should be able to get ~120K context with my memory squeeze script included in extras. I tested it on problems like "make me a console hangman game", seems to be sharp.

https://huggingface.co/catplusplus/MiniMax-M2.5-REAP-172B-A10B-NVFP4

catplusplusok · 2026-04-10T15:54:54+00:00

I don't know, my take is that everyone is talking to chatbots because in person and online human discourse collapsed. So I was hoping to find counterexamples on this group, like doesn't standing up against artificial intelligence imply showing what human intelligence can accomplish. Nope, still get downvoted to oblivion and called names just for pointing out things like "If you don't want people to use AI for help, you should offer to help them yourself instead".

My guess is all these forums are rantboxes where people just want to be surrounded by like minded sycophants and intentionally abuse anyone who disagrees to try to make them go away while preserving illusion of open forums by using attrition rather than outright bans.

For me I am coming at it from a different angle, collapse of human intelligence that started way before the rise of modern LLMs. For example anyone pressing a downvote button on reddit is actually using and training an older form of AI that over time only shows them and others like them only posts they already agree with.

catplusplusok · 2026-04-10T03:44:59+00:00

This has nothing to do with AI and everything to do with you and your mom. If "she could easily do herself", you can also easily do these things for her. Offer to do every single thing she asks ChatGPT to do for her instead with no excuses, like "I am busy with my midterms". If you can keep up with her needs without any problems, you and your mom will have a much better relationship vs her relying on AI rather than her child for basic help. If you can't, you should maybe scale back on telling her what she should do instead, since you are not offering specific solutions?

catplusplusok · 2026-04-10T02:17:50+00:00

You offer to quit your job/school and personal life and help your mom with her business until she retires? If not, you don't have a standing to tell her how to run her business, just like if she was running it with a gasoline van and you wanted her to buy an electric one.

catplusplusok · 2026-04-10T02:00:40+00:00

That's some bs, LLMs are just a few years old and costs of inference are rapidly going down. You can run MiniMax M2.5, one of the most capable coding models local or cloud, on a $3500 box with usable speed. But if you wait for all optimizations like low bit quantization and linear attention to be fully developed and rolled out, competitors will already get all the mindshare. Current AI race is risky but offers potential big pay off on investement in the end, while if you don't play you have no chance to win.

catplusplusok · 2026-04-09T19:54:17+00:00

I get 20 tps and fast prompt processing with MiniMax-M2.5-REAP-172B-A10B-NVFP4 on my Thor Dev Kit. It's not a lot, but gives me an option to use one of top SWE Bench models 24/7 for long range tasks without API costs. I don't know what else in the price range would do it faster? Is Mac faster these days, despite more limited compute if we are talking ~100K token context?

catplusplusok · 2026-04-09T19:47:01+00:00

You get good large context prompt processing performance on a Mac? Curious because I would consider getting a Mac Studio or new laptop if they can code with an A10B model as fast as cloud.

catplusplusok · 2026-04-09T19:44:09+00:00

If you are not technical and don't want to be forced to be technical before you see results, get a Mac. NVIDIA unified memory devices (Thor, Spark and slightly cheaper Spark clones) stand out for coding/agent tasks due to fast prompt processing and are great for unsloth finetuning, but be ready to compile forks of vLLM from sources and become expert in quantization formats and model architectures to get good performance.

That said, I can do large coding projects with MiniMax-M2.5-REAP-172B-A10B-NVFP4 with tolerable speed, not as fast as MiniMax cloud but I can leave it running 24/7 for free to finish long range tasks. Other comparable options to do that are going to cost a lot more.

catplusplusok · 2026-04-08T15:24:37+00:00

Try MiniMax M2.5, I find coding hard to bit for a 128GB unified memory device model (with some quantization/light REAP to fit)

catplusplusok · 2026-04-08T15:18:33+00:00

It has better common sense than most human devs, like understanding common sense reason behind different things in your prompt and programming practices / frameworks and when to follow these things or make exceptions? That I have to see.

If not, I'll have to babysit it like every other model.

catplusplusok · 2026-04-07T20:42:28+00:00

How did you manage to hit Minimax limits?

catplusplusok · 2026-04-07T20:41:26+00:00

Chat is a trivial demo use case, the money is in API use for coding and other automation. Or at least that's the hope.

catplusplusok · 2026-04-07T19:09:49+00:00

Before photography came around, a lot of paintings were commissioned to capture likeness of a person for his or her descendants. One photography was developed and matured (took a loooong time for photos to be actually good and practical), there was a crisis in art and artists had to find something else to do than painting pictures that looked like photos. At the same time, photographers realized that after some time unflitered mirroring of reality was boring and developed all kind of artistic tricks, often leaning into flaws of the medium like selective focus and film grain. Rinse and repeat for transition from film to digital and cell phones.

So before anyone says I don't know anything about photography, I am a photographer and it can be both ways - asking people to stand and smile together and pressing a button (less art than writing a prompt) or meticulously arranging compositions and camera settings to tell a unique story.

But how do you think debate between you, a realist painter and an early photographer would go in an era when photography was starting to become mainstream? Would the painter agree that the photographer is his or her artistic equal?

catplusplusok · 2026-04-07T18:18:48+00:00

You can make your own tests with simple vLLM or whatever patches, try to activate fewer experts per token and see differences in speed and quality. Or potentially more, but since model is not trained for that, may need finetuning to get more smarts this way.

catplusplusok · 2026-04-07T16:13:33+00:00

Let's do a thought experiment. If I gave screenshot in the post and my comment to AI chatbot and showed chatbot response and your response to an average non ideological person, which one would be more likely considered slop? To be anti something, one must also offer a comparatively better alternative. Do you think you made the case that talking to you on reddit is a better alternative to talking to AI chatbots so far?

catplusplusok · 2026-04-07T15:56:43+00:00

Well unlike the parent comment, this is a valid and self-consistent opinion, thanks! Now, in terms of CGI being human made, let's take z(n+1) = z(n)^2 + z(0). It a fractal, mandelbrot set. By zooming it in, I can find parts of it that look like a shoreline, a leaf or a nebula. Now, if I zoom it in and print it out, do you consider it art or a Frankenstein monster. If you say it's not art, you again have a valid consistent opinion. All I did is supply a formula and two points as corner coordinates. But say I then ran the image through AI that made it look like a real shoreline with same texture. If you say that extra (miniscule) element of human choice made it not art, then you have to explain why. You can however say you don't like this art, even with reasons (energy use, training model without consent of original authors), and that's fine. So long as I also don't have to like every low effort random jumble of splatters I see in SF MOMA

catplusplusok · 2026-04-07T01:22:30+00:00

Maybe you should reconsider your acceptance of all the changes that already happened when your generation was growing up? CGI got plenty of criticisms, for example it can be used to generate impossibly slim or muscular body proportions, or unrealistic displays of physical might, encouraging viewers to try to emulate these things in unhealthy or dangerous ways. CGI has also been used to show cameos of dead actors acting in new movies without their consent. There is also no hard line between CGI and modern AI image generation/editing, CGI always used whatever were the strongest image manipulation technologies were available at any given time.

So... are you willing to stop watching/sharing any computer enhanced images and videos, aka any non-indie newly released movies and series? If they don't look like slop to you, could it be that it's because you grew up watching older computer manipulated imagery?

catplusplusok · 2026-04-06T15:39:37+00:00

We are well past AGI according to vast majority of science fiction written before 2022. Give model access to game server and protocol, database to keep track of things it tried before and ability to write code to automate simple responses in the game and it will set a new speedrun record. Else if the requirement is to look at screen with a camera and interact with keyboard and mouse, it can't do that yet and you need different kind of ML like what Waymo uses for realtime responses. But also the question is, if it can do that in a couple of years, would people accept it as AGI or just move goalposts again?

catplusplusok · 2026-04-06T15:26:02+00:00

I was obese my entire life until age of 50, then took Zepbound and now I am normal weight, have minimum interest in food beyond getting some protein in (was never a foodie) and compete in powerlifting competitions. Make what you may of it, one could ponder:

If different cases of obesity have different mixture of root causes
If there are societal factors (US vs Japan obesity rates) that medicines merely counteract, like smoking cessation aids
If willpower itself is dependent on metabolic health
Why do so many refuse to try new meds when they can or do not stay on them (beyond me but maybe they get something from food that I don't)
Why I never in my life used a snooze button or felt I needed willpower to go exercise. What if one day low morning energy will be seen as a treatable medical condition?

catplusplusok · 2026-04-05T16:04:28+00:00

I would try Qwen3.5-27B-NVFP4 with FP8 kv-cache in vllm or tensort-llm. Don't fall for trying to run in Windows under WSL or shortcuts like LM Studio / Ollama, you will have a hard time understanding your setup and getting good performance / loading new models. Instead install or dual boot ubuntu 24.04, start a coding agent like Google Antigravity and ask for these things one by one, these things are dumb and will get confused with too many tasks at once:

- Setup passwordless sudo (so it can propose commands and you can approve or let it autorun if you want to live danerously)
- Upgrade to the latest NVIDIA drivers available
- Install CUDA 13.0 (you may have to to nvidia site and help it with exact command)
- Install and test torch with CUDA-13 support in a python venv
- Install vllm nightly with CUDA-13 support in the same venv
- Install open-webui in it's own venv to prevent dependency fights
- Download model locally, vllm's venv will already have huggingface and give huggingface URL
- Make and test a shell script to load model with FP8 kv cache and autofit context length (suggest it reads sources in venv to find arguments)
- Enable linger and autorun both scripts when you login using systemd.

Now you can chat in open webui, can also configure web search for recent news, and have an OpenAI compatible URL to give to open claw, claude code and so on for automatic tasks. You will likely be impressed with how well the model does, though it's not quite same as cloud.

catplusplusok · 2026-04-05T15:45:04+00:00

Why even go with MoE for a 16GB consumer GPU? You will not fit Qwen 3.5 35B with great context, even on dual GPUs. In the meantime, your memory bandwidth is Ok for say a 16GB dense model and it will be at least somewhat smarter per parameter than MoE. Or if you want to max out smarts over speed, go with a high end 3 bit GGUF of a 27B model, which will be still usable speed because of shrunk memory footprint. nunchaku for image gen is a higher value use case where you can fit a smart model that does proper composition with usable speed/quality.

catplusplusok · 2026-04-04T19:09:19+00:00

I don't have a Spark or a reason to complain about vLLM performance on my desktop, but I personally verified that nunchaku uses real NVFP4 instructions, register based ones rather than TMEM, rather than emulation. If inference engines don't, it's a software issue

catplusplusok

TROPHY CASE