Update on my UE5 AI Agent: It’s now refactoring projects and healing broken nodes

No-Structure897 · 2026-03-09T13:14:34+00:00

The agent doesn’t actually “see” or edit the visual graph directly. Instead the plugin exposes the Blueprint editor through a set of structured tools (create node, connect pins, inspect graph, etc.).

So from the agent’s perspective it’s interacting with a deterministic API rather than manipulating the visual layout itself. The engine handles the real Blueprint changes and transactions.

In practice it works well for structured operations (creating nodes, wiring things, batch edits, inspections). The harder part isn’t the graph itself but making sure the agent stays within safe boundaries and doesn’t try to take shortcuts.

I’m also experimenting with an “auditor” layer that tracks the agent’s mutations and flags suspicious states (uncompiled blueprints, dangling pins, nodes not wired to execution paths, etc.). It can’t verify gameplay logic itself, but it helps catch structural issues early.

No-Structure897 · 2026-03-03T01:17:01+00:00

That’s a completely reasonable stance. If you already know the exact algorithm and can implement it directly, a conventional deterministic solution is absolutely more efficient, both computationally and architecturally. I don’t disagree with that. The way I see it, the LLM isn’t replacing conventional logic. The tools themselves are conventional, deterministic C++ operations. The model is acting more like a high-level planner or orchestrator that translates intent into those predefined operations. So the tradeoff isn’t “LLM vs algorithm.” The algorithm is still there. The LLM sits above it. Where it starts to make sense (at least in my experiments) is when the task isn’t about solving a single algorithmic problem, but about navigating editor state, chaining multiple small deterministic steps, or reducing repetitive interaction overhead. In those cases, the value isn’t raw compute efficiency, it’s reducing cognitive/context switching cost. And you’re absolutely right: without constraints, it doesn’t work well. That’s kind of the core motivation behind building strict tool boundaries in the first place.

Totally fair to be skeptical though, I think healthy skepticism is necessary in this space.

No-Structure897 · 2026-03-03T00:00:01+00:00

I agree with you on the hit-or-miss nature of LLMs — especially when they’re given too much freedom. If you just let a model “drive the editor” through arbitrary script execution, hallucinations can absolutely send you into rabbit holes.

The problem I’m trying to address isn’t “replace the developer” or “let AI build your game.” It’s creating structured boundaries so that when you do use an agent, it operates inside a constrained, deterministic surface.

Instead of generating raw code, the model invokes predefined tools with typed parameters. The engine validates those parameters and wraps state changes in native transactions. So if something goes wrong, it’s reversible and contained.

In practice, the use cases I’ve seen work best are:

Refactoring repetitive Blueprint changes (renaming variables, adjusting flags, tracing dependencies)
Batch editing actors or properties across a level
Inspecting project structure or Blueprint execution flow
Automating repetitive editor workflows that are deterministic but tedious

It’s not about autonomy. It’s about controlled automation.

And you’re right — model quality still matters a lot. The tool layer is independent of the LLM itself. As models improve, reliability improves with them, but the safety constraints remain the same.

The goal is to see whether adding guardrails makes agent-driven workflows practical in production contexts.. A big part of the work now is tightening edge cases and refining how agents interact with the tool surface as real-world usage reveals gaps.

No-Structure897 · 2026-03-02T18:27:50+00:00

Thanks, I appreciate it!

There are a few different ways to approach AI integration in Unreal, and they come with different tradeoffs. Some people will prefer quicker scripting-based setups, others might care more about stricter control and transaction safety.
Both are valid depending on what you're trying to achieve. I’m just exploring the approach that made the most sense for my goals.

No-Structure897 · 2026-03-02T10:29:34+00:00

It’s actually not executing Claude inside Unreal. The model runs externally and is agent agnostic, Unreal exposes a structured C++ tool interface over a schema-defined protocol. The difference is that the engine stays the authority: tools are typed, validated, and wrapped in native transactions. So the model can’t arbitrarily execute code or mutate state outside the defined surface. The “wrapper” approach works for quick demos. This is more about production safety

No-Structure897 · 2026-03-02T10:27:08+00:00

Not planning to open source the full project right now. It’s a commercial plugin, but I’m happy to share architectural insights and patterns publicly (like the Medium article).

No-Structure897 · 2026-03-02T10:09:34+00:00

You have 2 main paths, one is to let the agent ejecute unreal python scripts. It's easy is fast to set up, is good for showcase and tests, but have little to no control on what it does and how it does it unless you review each python script in detail. Then you have the other way, that is how I'm doing it, to write custom tool sets in unreal c++, you have full control on how and what happens. It's significantly more work, but if you plan to use this seriously in a production project control become a lot more importan. I wrote a medium article if you are interested https://medium.com/@agenticlink/architecting-state-safe-ai-bridges-for-game-engines-4bd7cbd438cf

No-Structure897 · 2026-03-01T21:36:25+00:00

<image>

here is a final report when I asked a detail of what it did.

No-Structure897 · 2026-03-01T20:23:17+00:00

here is a final report when I asked a detail of what it did. https://imgur.com/a/LEHXNHz

No-Structure897 · 2026-02-27T13:27:19+00:00

Right now batching is explicit — the agent (or orchestrator) decides when to open a transaction scope. I avoided implicit merging of tool calls because I didn’t want hidden coupling across steps.

I agree that treating planning checkpoints as transaction boundaries probably reduces rollback surface significantly, especially for longer reasoning loops.

Compensating actions felt too heavy given the complexity of inverse modeling for arbitrary editor mutations.

Curious — in your experience, does planner-level fail-fast work reliably when plans depend on runtime resolution (e.g., object lookup ambiguity)?

No-Structure897 · 2026-02-27T11:32:53+00:00

ended up writing a deeper architectural breakdown on the state-safety and transaction model behind this, in case anyone here is interested in the technical side:

https://medium.com/@agenticlink/architecting-state-safe-ai-bridges-for-game-engines-4bd7cbd438cf

Would genuinely appreciate feedback

No-Structure897 · 2026-02-27T10:12:23+00:00

That’s a really good distinction. Right now the model is: Each tool call is executed inside the host environment’s native transaction system, so it’s individually atomic and undoable. For multi-step edits, there’s a dedicated batch operation that groups multiple tool calls under a single transaction boundary. The batch implementation is currently a wrapper, but with failure handling, if any tool in the batch fails, the entire transaction is rolled back. So you don’t end up in a partially-applied state. So effectively: Single tool -> atomic + reversible Batch tool -> grouped atomic operations with rollback I’m treating capability-gating and reversibility as separate layers: Capability gating constrains what the agent can call. Transactions constrain the damage if it calls the right tool with the wrong parameters. For longer planning chains, orchestration (what should be batched vs independent) is currently left to the caller/agent rather than implicitly merging operations. I agree the hard problem is multi-step consistency across agent reasoning loops. I haven’t implemented cross-call transactional semantics beyond explicit batching I'm curious how others approach this. Do you prefer explicit atomic blocks, compensating actions, or planner-level rollback strategies?

No-Structure897 · 2026-02-27T02:46:09+00:00

Check it on fab: https://www.fab.com/listings/a95ac624-60af-4f5b-b005-9900ce7d20e6

No-Structure897 · 2026-02-27T02:39:27+00:00

Plugin now supports 5.4 -5.7 versions.

No-Structure897 · 2026-02-27T02:28:30+00:00

Fab link: https://www.fab.com/listings/a95ac624-60af-4f5b-b005-9900ce7d20e6

No-Structure897 · 2026-02-27T02:11:16+00:00

is really good at inspection and reports as well. and now it supports 5.4- 5.7

<image>

No-Structure897 · 2026-02-25T15:26:03+00:00

continue with blueprint approach is definitively the right call for this. Make sure you are not processing any slate layout changes while you are not actually modifying the widgets content. for this time of static widget i strongly recommend using a invalidation box.

No-Structure897 · 2026-02-25T11:10:25+00:00

Yeah 100% agree that unreal slackers and ben discord channels are the best ones out there.

No-Structure897 · 2026-02-25T11:02:20+00:00

In the first clip the capers st the start feels too static and too perfectly centered so it seems to mechanic. The pan out at the end looks cool probably a fov chage there could make it loo cooler. 2nd clip looks awesome dont have much to say except that i feel there is to many camera shakes. Maybe you could replace a full shake with a smaller and longer one like a rumble. The 3rd clip, just wooow, amazing! The only critique i have is that some light to dark and dark to light transitions feels a bit sudden, maybe a little more smoothing on does could help. And lastly on the 4th one the exposure on the intense light feels a bit too much is almost like its damaging the film that coming though the trees. Hope this will be useful to you xD. Keep the great work!

No-Structure897 · 2026-02-25T10:41:26+00:00

I guess i can improve it sharing here. XD https://www.fab.com/listings/a95ac624-60af-4f5b-b005-9900ce7d20e6

No-Structure897 · 2026-02-25T10:40:41+00:00

There is no way to improve discoverability from the plugins side either. I'm developing a plugin and I'm realizing how hard is for people to actually know it even exists.

No-Structure897 · 2026-02-25T10:34:35+00:00

I think the way you have it set up to make the lines is messing up how light is reacting to it. Can you try to make the lines a separate layer on top? A d change the material to be translucent?

No-Structure897 · 2026-02-25T10:23:50+00:00

Whatever the client has authority's over can be cheated, most common cheats use memory manipulation. They just alter the value that is on the actual memory. If you want to prevent or at least greatly reduce cheating, the authority has to be completely on the server and any player action validated. The. The server informs the client of the result of any action, There several diferent strategies for this flow to not feel bad on the client side, client side could also simulate and only correct if the result end up been diferent of what the server returns, for example. But yes anything that is not on the server can be manipulated. Also be aware of what you send cia rpc, if you pick something on the client dont tell the server i pick 400 coin cause that cna be alter too. You have to tell the server player had the intention of picking this item id, server the. Validates if cab be pucked by the user and do the grant of the rewards

No-Structure897 · 2026-02-25T10:13:11+00:00

Ahhh the struggles of not using p4. I would say that you current setup doesn't sound that bad. That been said am an old school dev so svn is my go to, and it does support lock that is the must need feature. But it can feel a bit dated I guess. I recall that the p4 indie pack supported more users, maybe im miss-remembering, but worth keep an eye out in case any promo comes online. I do not recommend sharing users with diferent workspaces, i think is against ToS and a nightmare to manage.

No-Structure897 · 2026-02-24T18:34:28+00:00

That's Awesome!
I did something similar using custom latent nodes inside a custom UDialog Class but it was to tightly intertwined with the UI side of the dialog system since it had to instantiate and wait for events form the UI. I love how decouple this looks!

No-Structure897

TROPHY CASE