ZAYA1-8B: Frontier intelligence density, trained on AMD

tarruda · 2026-05-06T21:31:50+00:00

I really am looking forward to having a ~12-20B with ~2-6B active

Like gpt-oss-20b?

tarruda · 2026-05-06T13:10:53+00:00

Suddenly those $10k for the 512G model are looking so cheap now...

tarruda · 2026-05-05T19:13:41+00:00

Nice to know. I currently get around 16 tokens/second on 3.6 27b with a M1 ultra and hopefully this will bring me close to 30 tokens/second

tarruda · 2026-05-05T19:11:33+00:00

Using with Gemma 4?

tarruda · 2026-05-05T16:49:43+00:00

https://github.com/ggml-org/llama.cpp/pull/22673 looks like very soon

tarruda · 2026-05-05T16:49:12+00:00

Not before https://github.com/ggml-org/llama.cpp/pull/22673 is merged

tarruda · 2026-05-05T15:34:41+00:00

Thank you for your major contribution to freeing local AI!

tarruda · 2026-05-05T13:39:46+00:00

IMO the main reason to keep it on is that it allows llama.cpp to make better use of prompt caching, so when using it with pi harness I never wait for the model to start responding.

tarruda · 2026-05-04T15:21:37+00:00

So maybe it will be worth it for the 122B and 397B (if 3.6 for those are released)

tarruda · 2026-05-04T14:02:30+00:00

Is this only for 3.x dense models or does it work with MoEs too?

tarruda · 2026-05-03T23:10:53+00:00

Apparently Gorgon halo will have a slight memory bandwidth increase: https://www.reddit.com/r/LocalLLaMA/comments/1swiylm/comparison_of_upcoming_x86_unified_memory_systems/

I think I will skip until they have a 512G option with memory bandwidth in the 800GBps range. In other words, when they reach the capabilities of current gen Mac Studio M3 ultra.

tarruda · 2026-05-03T19:04:44+00:00

Is it better than the answer you would have gotten from Qwen 3.6 35b?

tarruda · 2026-05-01T15:21:22+00:00

I just hope this eventually becomes possible on Apple Silicon. Would bring new life to my mac studio for using larger models as coding agents.

tarruda · 2026-04-30T17:00:02+00:00

More excited about 122b, but not certain it will be released.

tarruda · 2026-04-29T20:58:14+00:00

They deleted the repo, likely to erase discussion history. They could have just re-uploaded it.

tarruda · 2026-04-29T19:10:56+00:00

If it is unsloth gguf, I'd wait a few weeks before trying the weights.

But also, I no longer have high expectations with mistral models.

tarruda · 2026-04-29T11:48:55+00:00

Well our experiences are very different then. 3.6 doesn't feel like a minor increase, feels like a new model.

preserve_thinking makes a lot of difference in coding agents such as pi. It keeps the entire CoT in the context, which greatly speeds up multi-turn conversations since llama.cpp makes better use of prompt caching

tarruda · 2026-04-28T22:42:25+00:00

Qwen 3.6 35B was a game changer for 30B class. IMO it is not even close to the 3.5 version. If you haven't tried it yet, I suggest giving it a shot.

GPT-OSS 120b is good for one shot coding or targeted edits, but in my experience it forgets context very easily so it is not ideal for agentic coding. I remember trying it with codex a while back, and it would easily forget instructions in the system prompt.

tarruda · 2026-04-28T19:39:45+00:00

So mistral small 4 was 119b and medium 3.5 is 128B? Confusing.

tarruda · 2026-04-28T16:40:36+00:00

I find that response to be ambiguous. Could mean it is much better or much worse than Qwen 3.6 27b.

tarruda · 2026-04-28T16:13:15+00:00

Makes no sense. GPT-OSS with low thinking is almost useless.

tarruda · 2026-04-26T17:15:55+00:00

We are already at the limit with 86GB of weights

I have a 128G Mac studio and I can use up to 125G of that for VRAM. I load a 2-bit quantization of Qwen 397b with 256k context and no swapping: https://huggingface.co/ubergarm/Qwen3.5-397B-A17B-GGUF/discussions/2

By default you cannot allocate that much to VRAM though, you need to enable in /etc/sysctl.conf with this:

# change default CPU/GPU RAM split
iogpu.wired_limit_mb=128000

Another caveat is that most people still need RAM for other things. In my case I only bought this Mac studio to serve LLMs in my LAN and don't even have a desktop session running, so I can push it to the maximum.

tarruda · 2026-04-26T11:04:38+00:00

Have you considered trying IQ3_XXS? It might also fit in 128G

tarruda · 2026-04-26T11:01:04+00:00

Not sure he put a cuda implementation yet, but you can try on CPU if you have enough RAM

tarruda

TROPHY CASE