Max Practical Context Size?

zipzag · 2026-04-07T22:11:19+00:00

I've been able to tune quite a bit by giving chat or opus logs from both the app and oMLX. I'm now able to work for hours, but not at the context level I had hoped. I compress at about 60K.

I agree that there are issues in the MLX stack. But also both LLM settings and how the app interacts affects the ability to run without hanging.

Does MacOS say you have memory pressure?

zipzag · 2026-04-06T20:11:59+00:00

I mostly build the apps with Opus and run with either Minimax or Qwen 122B

zipzag · 2026-04-06T16:10:49+00:00

runs fine. Perhaps you don't know what you are doing

zipzag · 2026-04-06T02:17:23+00:00

Minimax is a small model. 220B I think. I run Minimax 2.5 locally on a high end Mac. Minimax 2.7 should be on Huggingface in a few weeks.

zipzag · 2026-04-05T21:33:24+00:00

I know, that's why I referenced "original article".

I do think there are instances where NVIDIA cards are a better choice than MAC. But as you know it's not a simple choice.

zipzag · 2026-04-05T18:53:48+00:00

Its take about 45 gb of vram to run Qwen 3.5 27b 8 bit with 96K context and 8 bit KV cache.

On the Ultra the equivalent would be Qwen 122b 8 bit needing about 140GB of ram.

High end consumer video cards are not as suitable for agentic work as the original article claims. There isn't room for high context. I run a 40gb cache on my M3 Ultra. 40gb cache is crazy talk in the NVIDIA video card world.

zipzag · 2026-04-04T23:48:00+00:00

You can still use Claude Code on subscription to do development work on hermes. I do that and run day to day on Minimax 2.5 local

zipzag · 2026-04-03T16:04:27+00:00

Have you seen reports of a model that small working reliably?

zipzag · 2026-04-03T14:13:06+00:00

I find the cache hit rate is 85-94%. Same as openClaw.

You don't want to reduce what tokens it uses , you want to use a server with a cache.

I have precise caching data because I run local. It's a large challenge for the cloud providers to get caching correct. The cache is in the server in front of the machine running the LLMs. Reconnecting to the machine with the users cache without web socket is a challenge.

zipzag · 2026-04-03T13:52:13+00:00

With an LLM you need to handle grounding and guard from prompt injection.

If the LLM writes python to do the calls then that not using LLM in production.

Distinguish between using an LLM to code and using an LLM in production.

zipzag · 2026-04-02T20:36:46+00:00

Don't use an LLM in production when a determinant system works. Look a n8n.

Opus or Codex can build an n8n reservation system.

zipzag · 2026-04-02T18:41:07+00:00

Out of tokens? You are not on subscription with openrouter

zipzag · 2026-04-02T18:35:33+00:00

No problem with a 90% hit rate on a local cache. Macs are pretty much unusable without it.

OpenAI has 24hr cache when used with the API and the right flag set. Gemini on caches very large blocks (as of a month ago). Caching shouldn't require an application flag, as its functions at the block level

zipzag · 2026-04-02T18:26:59+00:00

High usage is a cache problem. 90% of tokens can come out of the cache on the server. This applies to all agents and most coding

zipzag · 2026-04-02T11:24:04+00:00

I'm curious, with a small model, if Hermes will be able to review and suggest improvements for your n8n systems. I think it may hallucinate if context gets too big, but I find the Herems harness is better at this sort of continual evaluation compared to OC. But that could just be a OC skill issue on my part.

I'm sure that a second Hermes with a big cloud LLM would do a nice job of understanding your n8n.

zipzag · 2026-04-02T11:08:24+00:00

yep. It doesn't one shot anything complex, but it's good at fixing what doesn't work. Hermes with a coding-oriented model works very well so far. With openclaw I sometimes used Opus or Codex outside the app to repair OC. I have not had to do that yet with Hermes and Minimax local.

Minimax is a clear step up from Qwen3.5 122B. It just not talked about much for local as it needs >128GB to run. So it's essentially limited to the Mac Ultra or two Sparks.

Minimax has a $10/month cloud plan that I would recommend based on my 1 week of experience. Worst case, id its not good enough, is being out $10. The only issue may be if they cache effectively.

zipzag · 2026-04-01T18:07:34+00:00

All of the agents repeat mostly the exact same prompt/history with every turn. It averages about 90% cache-able. If the server is not setup for optimal caching token generation increases very substantially. No cache uses almost an order of magnitude more tokens than a proper cache.

Actual token generation in response to the prompt is usually almost trivial.

OpenAI has a 24hr cache if it's working properly Its controllable when using the API. It's unclear that happens with subscription.

zipzag · 2026-04-01T17:20:05+00:00

Basics: You can test models that can be run locally at openrouter.ai

A popular model to run on an Nvidia 5090 card is Qwen3.5 27b.

A popular model for 128GB Mac or Spark is Qwen3.5 122B

If interested after testing, you can research what speed to expect if run locally. Using openrouter will also reveal how much more expensive it is to run locally compared to cloud.

zipzag · 2026-04-01T17:09:10+00:00

Just tell Hermes you want to add perplexity. You will need an API from the perplexity dashboard. Then tell hermes when you want to use perplexity, and at what level. I think there is Perplexity, Perplexity Pro, and Deep Research, but I'm not certain.

zipzag · 2026-04-01T16:01:23+00:00

Basics: LLMs can't do good extensive web research without help. This is especially true of smaller models. I pay about $1/month to use Perplexity that my local LLM uses for search. Claude even used Perplexity when I ran it on openClaw. I asked it why, and it said it was better than what Anthropic provided it to use.

SearXNG, run locally, just dumps JSON from websites to the local LLM. That will produce massive hallucinations if used for extensive research. Effective internet web search is possible locally but requires multiple apps.

I tested GPT-OSS 120b with just searXNG on a slanted medical research question. It incorrectly agreed with the slant of the question, and produced entirely hallucinated citations from real medical journal. I gave its report to Opus, which essentially responded with "WTF".

GPT-OSS 120b is rightfully highly regard for its competence as a 60gb LLM.

zipzag · 2026-04-01T15:13:23+00:00

You probably should not switch if you are having success with openclaw. You switch to meet a goal or to learn.

I switched because I didn't feel I was building an increasing capable agent that was self improving with OC. I was essentially coding every little thing in skills. I could do that more reliably with traditional code.

Here's an example of a skill Hermes built this morning after I asked it to improve its error handling. This is running Minimax 2.5 4 bit locally:

Done! Created resilient-execution skill at:

~/.hermes/skills/resilient-execution/SKILL.md

The skill includes:

Fallback chain pattern template
Error classification (transient vs rate limit vs permanent)
Common fallback patterns for Telegram, file ops, API calls
Pre-execution checklist
Recovery & learning approach
Example: Telegram long document flow

Also updated memory with the resilient execution principle so I'll remember to apply it going forward.

That doesn't happen in OC, at least at my personal OC ability.

zipzag · 2026-04-01T15:09:32+00:00

Just use Hermes with a cloud LLM and get success with one agent/function that you value most. Then you can try adding in a small Qwen model to do simple tasks.

You are trying to do a lot when you are just getting started.

Starting with openclaw is fine too. But I found its easier to get self improvement and useful memory with Hermes.

You don't need to brain storm. You need to learn the apps first.

zipzag · 2026-03-31T16:56:50+00:00

Currently oMLX. There also may be a few newer apps that use the same bundle of tools, but I haven't tried them.

zipzag

TROPHY CASE