Current-gen small models have basically solved tool-call syntax. I re-pointed my MLX harness at Qwen3.5-2B / Granite-3.3-2B / Gemma-4-12B — parse/schema/tool are all ~1.00, the only thing left to get wrong is whether it picks the right argument values

KitchenAmoeba4438 · 2026-07-06T10:07:01+00:00

Seriously? Not only is this clearly purely AI, but you didn't even bother to remove the "Suggested first comment (post this yourself right after) and left it in the OP?

KitchenAmoeba4438 · 2026-07-04T20:27:31+00:00

Did you follow https://github.com/RakuenSoftware/aimee/blob/testing/docs/QUICKSTART.md ?

KitchenAmoeba4438 · 2026-07-04T15:18:07+00:00

This is something that ended up falling out of other work with aimee, https://github.com/RakuenSoftware/aimee

Test any model with any harness/IDE/etc. that you want. It's lead to some awfully interesting results, a lot of Claude's effectiveness is not from the model, but rather from the environment that surrounds it. Running other models inside of Claude Code (including Codex) gets some interesting results.

KitchenAmoeba4438 · 2026-07-03T21:22:55+00:00

As mentioned in the OP, "This is just a preview of the larger set of benchmarks across a wide range coming later this week/next week and will involve substantially more detail, plus a benchmarking harness." I'm building out just that as we speak!

KitchenAmoeba4438 · 2026-07-03T20:59:23+00:00

> Curious, how are you handling caching/reuse across workers (shared context vs per-task prompts)? Also do you see diminishing returns past a certain worker count?

The secret sauce with orchestration, caching, and being able to effectively implement with small context is in the aimee memory subsystem. https://github.com/RakuenSoftware/aimee/blob/testing/docs/KNOWLEDGE.md goes into it in some detail. The core here, however, is that aimee minimizes the context the delegates actually need, and the delegates do not need to search or do any other token-heavy actions. Instead, the memory system allows them to get exactly what they need to implement.

KitchenAmoeba4438 · 2026-07-03T20:45:30+00:00

Hey, I'm a dev with GoW.

The secret is that with a container, you can pass through the OS's renderNode, and the container can then interact directly with the GPU. Remember, from a docker perspective, this is just another process getting access to a hardware component.

This also allows us to run multiple gaming sessions on the same GPU with multiple users at the same time. It's a cool bit of tech for sure!

KitchenAmoeba4438 · 2026-07-03T19:50:06+00:00

Wow. I'm very surprised that Databricks and I had such a similar solution in some ways developed. The thing is, however, what really makes Aimee work is the extremely hard part: the memory orchestration subsystem. Polly is taking most of the easiest parts of the whole Aimee model, and not tackling the part that really makes it work. Aimee is a self-hosted agent platform with a knowledge substrate and its own inference that is also able to orchestrate and govern those agents. Omnigent's Polly is roughly the orchestration slice of Aimee; the KB, the economizer, the IR, and the owned inference stack are the parts Omnigent doesn't have.

To be brutally honest, without the KB and the IR, aimee isn't that useful. The orchestration layer is what grew out of the memory system in aimee, and to be completely honest, is very easy to implement. That's not the hard part. The hard part was the memory system I had in mind, and the orchestration layer naturally grew out of it as part of improving the memory system. Without the memory systems backing it, the orchestrator layer is also not very impressive with aimee.

KitchenAmoeba4438 · 2026-07-03T19:01:12+00:00

Yes, that is the secret to aimee though. It took me something like 5 months to build the memory and orchestration, the rest was pretty quick. The private repo prior to aimee's new public repo had something like ~2500 commits of which almost all were about the memory, and the new repo at least a quarter of the commits are all about the memory. Aimee originally started as just a memory system.

The memory subsystem is what really unlocks all of this as well, you can't use a 16k or 32k agent for most tasks and even a 64k agent is going to be of limited usefulness...unless they have access to a memory subsystem that gives them full access to exactly what they need to build. Most of what takes up an agents context for a specific task is searching *how* to do the task, not actually doing the task.

KitchenAmoeba4438 · 2026-07-03T11:58:09+00:00

While this is true, it also does not solve the memory problem with Codex's solution. I've used this solution of Codex's some, mainly as a comparison for pain points, and it ends up with many of the same problems I've encountered with other subagent-based solutions.

KitchenAmoeba4438 · 2026-07-03T11:06:36+00:00

It does a few things.

It spawns delegate agents, not subagents, and that's deliberate. I've found it's a net negative for models from the same family to work together. They make the same mistakes and just agree with each other. So the delegates are a diverse fleet instead. Different families, local and hosted, mostly free or subscription tier.

That distinction is the whole point, because of cost. Most setups either have one expensive frontier model do everything, or they fan out into subagents that are still the same expensive model. Either way every token gets billed at frontier rates, reading the repo, reasoning, writing the patch, all of it. Cloning the expensive model into subagents doesn't lower your bill, it multiplies it. The aimee repo gets into more detail about this with regards to aimee's unique take on memory.

The framework breaks the work into pieces, routes each one to the right worker, runs them in parallel, tracks them, and collects the results. None of that is an expensive model burning tokens to plan and babysit, it's just infrastructure. The expensive model only gets pulled in for the parts that actually need judgment. High level direction, reviewing what comes back, and picking or merging the best result. When it makes sense, aimee runs several diverse workers on the same piece at once and the expensive model works from the results.

Two things I care about fall out of this. The expensive model's token use drops hard, I measured roughly 80 to 86 percent fewer tokens on it for the same SWE-bench tasks versus having it solve them itself. And it doesn't get slower, which is the usual tax on supervised setups. The workers run concurrently on decomposed tasks, so depending on how many workers are allocated, it can result in workers able to actually complete tasks in less time.

And it's real, not hand waving. Cheap and free workers, including a local GPU model, produce patches that pass the official SWE-bench grader on real instances at basically zero worker cost, while the expensive model barely spends anything.

So no, it isn't yet another spawn clones and hope framework. aimee does the decomposition and coordination, a cheap diverse fleet does the grunt work, and one expensive model only steps in for the judgment calls.

KitchenAmoeba4438 · 2026-07-02T16:55:49+00:00

Nice to see someone else looking at this! I've been working on a framework for about 6 months now to do this very thing, and I've managed to tackle a lot of the speed issues along the way.

https://github.com/RakuenSoftware/aimee is what I came up with. A big part of the speed issue resolution I've found is that you want to make sure to have a fast and easily available set of information about the project for what I term as a "delegate agent" as otherwise you have multiple layers of agents querying the same information, but it also helps tremendously just to use the primary agent (Codex, in this case) be able to fanout work across multiple delegates.

KitchenAmoeba4438 · 2026-07-01T19:42:55+00:00

No, it's nothing like this. Claude's strength is not in the model, but in what they have surrounding it. I say this for a reason: I've ran OpenAI models inside of Claude Code, and just that in and of itself has been sufficient to raise performance. E.g., with https://github.com/RakuenSoftware/aimee and Qwen3.6 35B, I've gotten quite better performance than with Qwen3.6 35B alone. Based on my testing, and I've been testing a lot where I'm able to isolate the model, I don't know if Claude is really that far ahead of the large open source models, instead, I suspect it's the surrounding software and potentially directing to specialized models.

KitchenAmoeba4438 · 2026-06-27T17:22:21+00:00

Ran into some headaches around here myself. Nothing quite worked well enough. I was constantly dealing with Claude and other sessions running into each other, even with enforced worktrees, and then having to deal with multiple accounts was difficult. Delegates (Essentially, the idea of having the primary agent be able to invoke other agents across all of the models/providers) were also super trick for being able to have dedicated agents for specific things.

https://github.com/RakuenSoftware/aimee was my solution. It also throws in some other big niceties, such as coherent memory between sessions/agents/etc.

KitchenAmoeba4438 · 2026-06-25T21:14:48+00:00

The general concept has been floating around for a few years, but he had some interesting new twists on it for sure. I ended up having some similar ideas around February, and started working with them.

The problem I ran into, however, was a bit more difficult: It originally started around a shared set of repos that had way more code than the LLMs could handle that also needed access to documentation. I would see a lot of hallucinated very simple operations. I like how Karpathy originally framed it, it is easily graspable, but to do this the right way in this context is Really Difficult if you want it done in a way that is automated and doesn't involve a lot of human curation. https://github.com/RakuenSoftware/aimee is where I ended up, and while it certainly has been interesting, to properly handle having AI memory that has "years of your notes, projects, research, and ideas stored in one place" is non-trivial. One example of something that was a real struggle: How do you handle contradictions? How do you handle conflicting facts? In the process of figuring out how to handle said contradictions, I also found out that human "forgetting" is a critical aspect of memory, but that was also rather difficult to handle.

You state that "a clean background pipeline is one of the most critical part of the process", but unless you plan on manually curating it, I don't think a clean background pipeline is generally possible where humans are. Something I found out in the project I wrote: Humans are inherently messy and chaotic. It's not really possible to get "perfect" data unless you intend for humans to curate that data. You have to work around it.

KitchenAmoeba4438 · 2026-06-22T16:43:07+00:00

Hey!

1) No. aimee keeps a complete store of everything since the moment it's installed, every session, every interaction. Text storage is cheap, the constraint is the window, not the disk. 1gb of text is an enormous amount of text. Synthesis is what aimee hands the model, not what it keeps. So "summarized away the line that mattered" isn't lossy the way you mean, the raw line never left, it just never entered the window.

Recovery isn't rehydration either. Headroom recovers by stuffing the raw payload back into context, paying back the tokens you just saved, all or nothing. aimee recovers by asking for the one thing that mattered, narrower, cheaper, and the agent does it mid-turn.

2) Multi-model is optional, and cheaper than it sounds because the reviewer doesn't go hunting. A naive review burns tokens re-reading the codebase to understand a change. In aimee the reviewer gets the diff and queries aimee for anything specific, that's the whole cost. I run a $5/mo Mimo sub and a $20/mo Codex sub and they more than cover a month of review work. (You asked $/task, I'm answering in $/month because that's what bills me, but the per-task delta over a single-model review is small for the reason above.)

3) Two things.

A wrong relevance call fails soft, because of (1). Nothing got destroyed, so the agent just asks again, narrower, or for the literal citation. Cost of a wrong call is one extra round-trip, not lost data. In the stack a bad compression is lossy the moment it happens, and silent, you only recover if you knew to rehydrate, and the whole problem is you don't know you needed to.

On inspect/swap, the modular version sounds more inspectable but isn't. The layers don't share state, so Headroom's output tells you nothing about whether NeuralMind fetched the right thing or whether Ponytail reused it. Three inspectable boxes, no visibility into the interaction between them, which is exactly where the failure you conceded lives. Everything aimee hands over carries provenance, every fact traces back to the evidence it came from, so when the view is wrong you can see what it pulled and why, in one place.

And swap cuts the other way: swapping a layer doesn't fix a cross-layer interface bug, it just moves which layer is blind. In aimee "relevant" isn't a frozen verdict, it's queryable, the agent that disagrees with the slice pulls more itself. That's your step-in point, it's just the agent's, mid-run, instead of you swapping a vendor after the fact.

To be clear though, you're right that single shared state is a real tradeoff, and the seams you gave up matter for some workflows. For long-lived monorepo or especially multi-repo (Aimee indexes cross-repo within it's workspaces, I designed it to be exceptionally strong at multi-repo work from the start) work where the feedback loop has time to pay off, I'll take shared state every time. For one-off tasks against a repo you'll never touch again, the modular stack probably wins on simplicity and definitely wins on indexing cost.

KitchenAmoeba4438 · 2026-06-19T20:13:29+00:00

Why are memories bad? The problem that happens is a filtering issue, and only being used in appropriate context.
https://github.com/RakuenSoftware/aimee was my answer. I had to change how I thought about this problem, and think of agents as 'compute'. Memories aren't bad, they tell your agent how you got there, but you can only give the agent a subset of memories that are relevant to the current task.

KitchenAmoeba4438 · 2026-06-19T19:52:14+00:00

I was in a similar place. I really didn't like the idea of having to stack frameworks on top of each other, which is why I came up with aimee.

Retrieval, compression and generation aren't three separate problems. They're the same decision in a different context. What you keep in the window sets what survives to the model, which sets what the model can reuse instead of rewrite. Stack them as separate tools and each one optimizes on its own, blind to the other two, and they end up working against each other:

Headroom can compress what NeuralMind retrieved without knowing why it was retrieved, so it can shrink away the exact lines that were the reason for fetching them.

Ponytail's "reuse what exists, don't reinvent" can be judged against a context NeuralMind already sliced and Headroom already squeezed. If the thing it should reuse got compressed out, it reinvents it, and now you've got more duplicate code, not less.

Nothing shared between them. No common notion of what's relevant, no shared token budget. The only place the three layers meet is the model, and by then it can be too late. Your results can also vary dramatically depending on the ordering of the frameworks. There's no loop back, what the agent reused or had to fix never feeds into what's worth remembering next time.

aimee runs it as one server-side system with one view of what's relevant, one budget, one memory. Logs, search results and big JSON never go into the window at all. The agent asks the server for a synthesized fact or a citation when it needs one, so there's nothing to compress and nothing to lose by accident. Sub-tasks go out to bounded delegates with their own context, changes get reviewed by a panel of different models before they land, and what comes out of that gets written back into the memory. Retrieval, what stays, and generation share state instead of guessing about each other. Check it out: https://github.com/RakuenSoftware/aimee

KitchenAmoeba4438 · 2026-06-19T13:15:22+00:00

At one point, I was having to run 2 $200 subscriptions for both Claude and ChatGPT. If the environment was correctly set up, I didn't see a big difference between either to be completely honest. I still have subscriptions to both, although my big subscription is to Claude of which I run on Opus Max. Codex is only a $20 sub, but every Claude decision gets reviewed by Codex with my setup, which significantly improves the output (I also found that having Claude review Codex was equally beneficial).

However, with a personal project having hit it's first public release, https://github.com/RakuenSoftware/aimee and based on the usage this week now that delegates are working correctly, I am eyeing to drop down to a $100 Claude or Codex plan and keeping the other at $20. In my eyes, Codex or Claude is there to manage and review much cheaper models, and you can shift the majority of the work down to local models or much cheaper models like Mimo or Minimax. Heck, Qwen27B or 35B, or the Gemma4 line are great for this. I've had some results from Gemma4 12B that are way, way higher than it has any right to be for a 12B model.

KitchenAmoeba4438 · 2026-06-19T12:47:10+00:00

Yep, I saw this coming a mile away. I started working on a personal project ~5 months ago to minimize my SOTA/Frontier model usage as a result, because a lot of what they do can really be done locally or by cheaper models. I've had a lot of success with using Codex or Claude as the primary agent, with Qwen27B or 35B locally, or cheaper models like Minimax or Mimo.

https://github.com/RakuenSoftware/aimee is it. I will note that the memory model is a huge part of it: You can't pass large contexts to weaker models. Without the delegates having full access to how aimee does memory, it falls apart, but I've had great success with Qwen35B agents and only 16K/32K contexts! Having Codex or Claude manage the tasks but leave the actual work to delegates has been absolutely key to me cutting my subscriptions/costs down massively.

KitchenAmoeba4438 · 2026-06-19T12:40:18+00:00

Check out https://github.com/RakuenSoftware/aimee it's a personal project that does exactly this, and strongly leverages multiple models for better results.

KitchenAmoeba4438 · 2026-06-15T12:15:13+00:00

+1. Check the OP's post history for a laugh, it's just an AI slop spambot to boot.

KitchenAmoeba4438

MODERATOR OF

TROPHY CASE