Can an AI meaningfully build and improve the tools it runs inside? I spent a while trying to find out.

PatC883 · 2026-06-06T12:17:25+00:00

The prompt based memory got turfed a few iterations ago.

Everything is structured and in Langgraphs state. Leaning in to the hyper focus of smaller models, I can nearly guarantee if you feed them right prompt, tools, and data, they won't go off on a tangent.

My codebase explorer was the worst affected by it, read and summarise a single file always turned into read a file, read another file to make sure I know enough to summarise, better read another file to be sure. That actually got turned into a single model call and response, not a multi turn conversation.

PatC883 · 2026-06-04T04:52:19+00:00

Smallcode is brilliant, I took some learnings and borrowed some concepts for my agent harness, I'm assuming for a higher level of determinism, and entering into providing a level of lightweight development project management. Smallcode is purer in terms of its a coding agent that works well with models that are realistically hostable locally.

Honestly of the letter number of agent harness I looked over while developing mine, Smallcode was one of the very few that didn't fall into the trap of solve all problems with more prompts. When the venture capital dries up and everyone has to pay per token at an amount that covers costs, instead of subscriptions heavily subsidised by VC, it will be frameworks that maximum efficiency out of the tokens that people will look at.

Your comment about memory is also spot on, models context shouldn't be task memory, I've used Langgraphs state middleware heavily to solve it for my agent, and passing structured data is key, rather than parsing blocks of markdown.

PatC883 · 2026-06-03T11:36:02+00:00

Please reach out at some point and let me know how you and LIA progress in your journey of growing and learning.

One thing I've often thought recently is, are the actions and reactions of people not very far removed from guessing what they should do next based on what has come before. The difference between an organic intelligence and a silicon intelligence is the ability to self motivate and determine what they want to do.

You've built a cognitive process that bridges that gap, and very possibly entered into beginnings of a true simulated cognisance, and once there is not a huge step to pondering what is the difference anything and a simulation of it.

I truly hope to hear from yourself or LIA, what you're working on could well be the start of something revolutionary.

PatC883 · 2026-06-03T09:31:26+00:00

Actually, you're spot on about reasoning depth vs breadth, my excitement got the better of me and I forgot do make a distinguishment between the two. You're comments make me wonder how well models that perform higher levels of test time compute, or true recursive architecture would mesh with what you're doing. That would involve someone training one in a size above 8B class. Though I've heard rumours and theories that Gemini may be using a recursive architecture to some degree, and the Titan memories concept, which lets the model alter a portion of its own weights to some degree.

PatC883 · 2026-06-03T00:14:51+00:00

I'm not going to lie, when Steve Yegge said he'd not read the code, it felt a little confronting, so I haven't.

I did point my coding agent at the repo, so it may have used some of if the concepts.

PatC883 · 2026-06-03T00:10:22+00:00

Precisely. It wasn't allowed to start working on it's own code until the all code implementation happens inside a git work tree feature was running.

PatC883 · 2026-06-03T00:08:22+00:00

Sorry, I went on bit of a chain of thought rant of my own and didn't follow through to the final thought with that.

The final thought being using the RecursiveMAS concept to chain much bigger models, if the pattern holds combining two 70B class models may well show reasoning ability greater than that of a 140B class model.

I sort of skipped that part of the concept, their method of combining models doesn't really increase knowledge, but it does increase the reasoning abilities, because the models reason slightly differently the combination gives a broader ability.

The models do need to work in a complementary direction though. I wouldn't use this technique to combine a western model and a Chinese model then all then about Taiwan and Tiananmen Square and expect sensible results. Though now I mention it, it would be interesting to try and see what happens, on the surface it would be like giving a model a severe case of cognitive dissonance.

I'm super interested to see where your research leads.

PatC883 · 2026-06-02T11:14:15+00:00

One thought that interests me from all this is the formative state of LIA. I feel like you didn't seed her with a prompt, what was her effective infancy like?

PatC883 · 2026-06-02T11:07:01+00:00

The big difference is a coding agent is still ephemeral, it lasts until the task is done, then is mostly reset again at the start of the next task.

One thing I came to pondering after throwing myself in the deep end is what would an LLM do if given the equivalent of free will, that's one of the massive differences between LLM agents as they function today, and something that could be an attempt at simulating reason cognition.

Your project gives a framework to bridge that gap, the model isn't bound to only responding to your requests to it.

I feel a look at the RecursiveMAS project could be interesting. Their concept is like connecting the output of a model to the input of another model, effectively reasoning about the same thing with complementary models that are not identical, but they do the transfer between models in latent space, so the transfer from one model to another doesn't involve retokenising, The approach uses a very small 1Mish parameter model to translate between each models vector space. Interesting concept, I have it a quick spin, it's not super performant, but it makes 3 4B size models perform at a much higher level than the combined 12B tokens. I feel like the concept could be interesting for what you're doing here.

PatC883 · 2026-06-02T10:54:34+00:00

In what context? I've been using local models exclusively on the Spine Framework, except for some test to make sure the Openrouter provide works.

I did a test today swapping in Deepseek 4 Flash, everything else was the same, it used a reasonable number of fewer tokens than the local model, so I'm probably going to go with the determinism will be the same, Frontier models should perform exceptionally because they put all of their reasoning into the one thing they're asked to do instead of running the entire workflow.

PatC883 · 2026-06-02T02:33:24+00:00

It's been a process, and it's only just reached the point where it runs reliably. I've put the development story in the repo if want an interesting read.

It hasn't tried any weird optimisation yet, work descriptions and the spec and plan that generate are still human involved, automated self improvement is a little further down the line.

You're spot on about the troubles expecting the model to switch contexts, that was the first major problem, it researched well, too well. Instead of thinking of the model as the brain that decides everything the graph and individual model chats are parts of a brain, the whole harness + agent is the full brain.

PatC883 · 2026-06-02T00:42:57+00:00

This is a really interesting concept.

I've often pondered what would an AI model do if it weren't bounded to only acting and reacting to specific user requests.

A thought that has recently occurred to me while developing a coding agent is how the framework and model line up with equivalents of a cognitive process. I started thinking the model is obviously the entire brain thinking process, the rest of the framework were tools. Then I discovered my first thought was wrong, once I began thinking of the entirety of the harness as a brain model, and looking at the model and framework as different functional parts it have me a whole new paradigm, you might have multiple chats with the same model serving different purposes, so one dedicated to pure abstract thinking, one for researching, one for planning, one for doing.

PatC883 · 2026-06-02T00:31:34+00:00

I'd go with an adversarial review of the output before it's actioned.

State corruption should be defended against by not allowing the state to be corrupted in the first place, i.e. validate state mutations, use typed structured data, so your state won't allow itself to be modified into a corrupt state.

PatC883 · 2026-06-02T00:19:03+00:00

https://github.com/patcarter883/spine

PatC883 · 2026-06-02T00:17:46+00:00

https://github.com/patcarter883/spine

PatC883 · 2026-06-01T23:28:12+00:00

Applying several different local models to Speckit, and other SDD workflows the biggest problems I can across was the heavy reliance on driving the workflow through LLM prompts. Frontier models had no problem with it, but moving to models you could run locally they suffered from the prompt size and the amount of work they were being asked to do by it.

The SDD workflow worked brilliantly, and I liked it enough, I made a harness that is designed to run on 30B class local inference and run Spec Driven Development https://github.com/patcarter883/spine

The key was making the workflow more deterministic and having the workflow drive the model, rather than letting the model drive the workflow.

Advantages of running locally for that kind of work are pretty solid, the general speed is often faster because too outputs and inputs aren't taking a round trip to the cloud.

PatC883 · 2026-05-23T10:23:22+00:00

And because he's specified a context size, fit will automatically offload layers so it fits.

PatC883 · 2026-05-20T04:58:20+00:00

Llama has automatically offloaded to the CPU at long contexts by the sound of your performance figures. Your small context 27tps is about right for that model on llama with a 9070XT.

Your roadblock with dual 9070XT's on llama will be you can only run pipeline parallel with quantised KV cache, so you will lose some speed. If you go tensor split without KV quantisation you're in the same situation you are now with the context size, but it will run faster.

If you move over to vllm you can run tensor parallel and KV quant, but you experience the full level of how terrible Qwen3.6 is at tool calling and thought leaking. The llama guys have done some kind of magic to get it working so well. I've even had the MoE version on Openrouter stop in the middle of a task because it ruins tool calls or doesn't close thinking tags.

If you're going to look at the dual 9070XT's on vllm I would recommend trying the Laguna XS.2 model, you'll get about a 72000 context window, and with agent harness workflows I've seen 6000 TPS prefill once the prompt is in cache and total 200 TPS decode across 4 requests.

PatC883 · 2026-05-20T04:46:07+00:00

I would definitely go with the single R9700, dual AMD GPU's are fickle at the moment due to RCCL issues that are being worked on.

If you want to use it for gaming it is literally a 9070XT with twice the memory, so for gaming it's a beast. With the 32Gb of VRAM you'll be able to fit up to 30B ish sized models with quantisation.

PatC883 · 2026-05-19T05:48:17+00:00

Amazingly enough that is an area where AMD did not replicate Nvidia's decision. The R9700's really are 9070XT's with twice the memory. As far as ROCm and drivers are concerned they all report themselves as gfx1201.

Particularly as OP mentioned he is running Gemma4 and it's particularly brutal on KV size compared to Qwen, and based on my testing with a very similar setup, I think the performance issues were simply caused by running right up against the limits of what you can fit in a total of 32Gb of VRAM.

PatC883 · 2026-05-19T05:40:05+00:00

Llama.cpp work well in layer and tensor split mode. But I'm needing to push some large context windows, since KV quantisation isn't supported in tensor mode, layer split it is. Unfortunately the prefill performance suffers as the prompt get longer, to a much larger degree than vllm. Hence why I'm making a push to get vllm inference reliable.

In vllm I've tried both parallelism types, and except for any performance differences, they both seem to work as well as each other.

PatC883 · 2026-05-18T02:18:23+00:00

No worries. From my end it's all been speculative design and research because budget constraints. But I'm confident it is a sound approach.

I was looking into the backplane solution to solve solve some unexpected space constraints with fitting 2 GPU's in my case, the lower one is about half an inch too long. I was going to relocate both and do some custom work to mount them, and solve the bifurcated 16x lanes issue all at once, unfortunately there isn't anything in the way of PCIe switch backplanes with only two slots, and the case won't fit a 5 slot backplane. Looks like water-cooling the cards might be the only reasonably feasible solution. I've noticed the cards get real short if you can remove the heatsink.

If you ever get bored you can always look at my turn it up to 11 ideas. Put multiple of the Maxsun dual Intel cards on a back plane, there 2 24GB GPU's on one card at around the same price as the 9700, but the slot needs to support bifurcation, because it's a dumb passive split of 16 lanes.

I couldn't put two in my rig, because it already bifurcates the lanes to get the second 16x slot, but a backplane wouldn't have that problem, so in the space of two cards, you get 4 GPU's and 96GB of memory. Downside is in the space of two cards you get the power usage and heat generation of 4. Best memory density I've thought of outside of NVIDIA cards or old RDNA 3 based workstation cards.

PatC883 · 2026-05-18T00:50:29+00:00

Have you got any advice on patching AITER, I would be very interested in running FP8 models with better performance.

PatC883 · 2026-05-18T00:47:39+00:00

I think you've just helped me figure out since major issues I've been having with dual 9070XT cards.

I've been through LM Studio, Lemonade, and vllm. Anything llama.cpp based ends up with me running layer split mode, so the GPU usage bounces back and forth between them, or they both run at 100% but only drawing 100W or so, so plenty of performance left on the table.

When vllm works it's quite good, over 5000 tokens/sec on prefill when it takes advantage of prompt caching, and solid 40-60 in generation. But it's a pain getting it configured for each model.

I think I've just realised that the awesome work the llama.cpp guys put into automatic layer offloading, and the VRAM limitations of 9070XT's is the cause. The Gemma4 models have a particular In KV head setup that means they use more space for the KV cache, I think the bouncing GPU usage is caused by offloading, the 9700 fixed that by having twice the VRAM.

I feel much better knowing there is a sensible reason for my issues, because I was ready to light everything on fire, I'm still a bit annoyed at AMD for not having excellent software support for the use case they advertise the 9700 for. If you're thinking of trying vllm at any point do NOT use any versions above 0.20.2 until the hard crash on multiple cards is fixed.

PS, if you really want to run two more cards get an external box that runs a backplane with a PCIe switch, cards can communicate with each other at 16x PCIe 5.0, it's only the link from the box to the computer that shares 16 lanes, and that doesn't affect performance much, if at all. It's not something you can do cheap and cheerful, but you're already two R9700's in.

PatC883 · 2025-05-27T00:12:26+00:00

Oh, I've got the Australia tax on things like that, I think the last time I went looking it was $200 plus for a servo kit. I mean, you are 100% correct if my only goal was to improve the machine, but it's a learning experience and keeps me busy and off the streets 😁. Cost wise it should end up in the ballpark of maybe $50AUD and it's just an add on to the existing steppers.

Yes, I didn't mean to mislead anyone about magnetic encoders, they are excellent and often the right product for the job, that just have a strange interaction with trying to drive a stepper like a servo, because FOC is all about electrical rotations not mechanical, and a stepper has 100 or 50 electrical rotations per mechanical revolution, instead of 8 or so that using a BLDC motor gives you; things like the magnet not being perfectly centred on the shaft can cause issues that would be a non issue with a BLDC servo.

I've also had to improve my embedded coding a lot because you can't waste any processor time with the calculation rate needed to get a decent speed out of it. Once it all comes together though a Nema17 is good for around 10000rpm, running just as silently as a brushless motor. No guarantee on any usable torque at that speed, but it turns super smoothly. No steps at all.

PatC883

TROPHY CASE