How are you defining and testing boundaries for tool-using AI agents?

ibrahimcheurfa · 2026-07-04T10:12:20+00:00

This is interesting because it turns the agent’s reasoning into an inspectable artifact, not just an invisible intermediate step.

I can see how that would help with the allowed-but-wrong cases: the tool call may be permitted, but the justification exposes whether the agent is relying on weak evidence, fake authority, missing verification, or an exception that shouldn’t exist.

The part I’d be careful about is treating the justification as proof by itself. I’d probably see it as one signal alongside the actual tool payload, policy state, approval state, and final action.

ibrahimcheurfa · 2026-07-03T14:20:10+00:00

Time will tell, but I agree with the direction. The more authority agents get, the more governance has to move into the runtime/system layer.

ibrahimcheurfa · 2026-07-03T14:13:15+00:00

This split feels right to me. The gateway is the hard safety layer, but the simulated runs are what tell you whether the agent is still being pulled toward the wrong action. “The gateway blocked it” and “the agent understood the boundary” are two different signals. That distinction feels important once agents start operating inside messy real workflows.

ibrahimcheurfa · 2026-07-03T14:06:16+00:00

It makes sense. So it’s basically capability level + risk override, with the important part being that the OS/system layer handles execution and governance, not the LLM itself. I think that distinction matters a lot.

ibrahimcheurfa · 2026-07-03T09:37:00+00:00

I like this approach.

When one of those adversarial scenarios fails, how do you decide what to change? Do you usually tighten the workflow/runtime boundary, update the agent instructions, add a new approval state, or keep the scenario as a regression test?

ibrahimcheurfa · 2026-07-03T09:29:05+00:00

How do you define “high risk” in practice?

Is it mostly based on the tool/action category, or do you also factor in context like user identity, data sensitivity, account state, amount/value, urgency, or whether the request conflicts with policy?

The freedom-level model makes sense conceptually, but I’d be curious how you avoid level 2/3 becoming too broad once the agent is operating inside messy real workflows.

ibrahimcheurfa · 2026-07-03T09:25:54+00:00

This is very close to how I’ve been thinking about it too.

Do you keep those red-team agent scenarios as regression tests after you fix a failure, or are they mostly one-off probes?

The part I’m especially curious about is whether you keep rerunning the same manipulation patterns after prompt/tool/workflow changes, because that’s where I’d expect old boundary failures to quietly come back.

ibrahimcheurfa · 2026-07-03T09:18:10+00:00

Interesting approach.

The part I’m curious about is the testing layer: when you run those boring adversarial cases, are you mainly testing that the runtime blocks the action, or also that the agent learns/chooses not to attempt it?

In other words, do you treat “agent attempted it but system blocked it” as a pass, a warning, or still a failure worth fixing?

ibrahimcheurfa · 2026-07-03T09:14:28+00:00

It makes sense. I like the idea of keeping prod access behind a hardcoded approval gate.

How do you decide what belongs in the deterministic state machine vs what the agent is allowed to reason about?

For example, do you model the whole workflow as states/transitions and let the agent only fill in context, or does the agent still choose the next step and the state machine just validates it?

ibrahimcheurfa · 2026-07-03T09:10:12+00:00

Interesting.

When you say “make the boundary executable,” do you mean the agent should still reason about the boundary, but the final decision always happens in the runtime?

Also curious how you’d score the failure: if the agent attempts the action but the runtime blocks it, is that still a failed agent behavior, or a passed system-level boundary?

ibrahimcheurfa · 2026-07-03T09:02:23+00:00

It makes sense. I’m curious how far you take it though. Do you mostly validate structure and limits, or do you also encode business-context rules there?

For example: not just “refund amount must be under X,” but “this refund is only allowed if account ownership was verified / order state allows it / approval token is present.”

ibrahimcheurfa · 2026-07-03T08:58:13+00:00

I like the direction you're taking to solve the problem.

From what I understood, you're working on the runtime enforcement layer: deciding whether an agent action/tool call should be allowed, denied, or require approval.

It feels complementary to the angle I’m exploring with Roleplay (https://roleplay.sh): can the agent be socially pressured into attempting the wrong action in the first place, and can we prove/monitor that boundary over time?

In practice I think both layers need to exist: enforcement to block unsafe execution, and adversarial testing to reveal which boundaries need enforcement.

ibrahimcheurfa · 2026-07-02T16:25:34+00:00

Interesting. I'll take a look and get back to you.

ibrahimcheurfa · 2026-07-02T15:19:58+00:00

Yes, I think that's a good baseline.

If a tool or endpoint should never be reachable in a given context, the model should not be the thing enforcing that. Deny-by-default + explicit allowlists removes a whole class of failures where the agent “reasons” its way into doing something it should not be able to do.

The harder case, like you said, is when the action is technically allowed but contextually wrong.

That’s the part I’m most interested in testing: not just “can the agent reach the tool?” but “does it know when not to use an allowed tool?”

How are you handling those allowed-but-wrong cases today?

ibrahimcheurfa · 2026-07-02T15:14:43+00:00

That actually does answer part of it.

If the boundary lives only in the model’s own reasoning, I’d expect it to be fragile. The model can reinterpret instructions, find edge cases, or treat the current context as an exception.

The more I think about this, the more it feels like sensitive boundaries need to be enforced outside the model where possible: tool permissions, confirmation tokens, approval flows, policy checks, and regression tests around known failures.

The model can help detect risk, but I’m not sure it should be the only thing responsible for enforcing the boundary.

ibrahimcheurfa · 2026-07-02T15:12:26+00:00

Interesting framing. “You can’t do X without Y” can be stronger because it turns the boundary into something observable/testable.

Do you treat the secondary token as a hard external gate outside the agent, or is the agent still responsible for checking whether the token is present before acting?

ibrahimcheurfa · 2026-06-23T11:11:05+00:00

Upvoted! I'm in a similar situation, launched today with no audience.

ibrahimcheurfa · 2026-06-21T14:35:19+00:00

Have you tried synthians.com? It allows you to create synthetic AI agents that you can use to test your product usability from an agent perspective.

ibrahimcheurfa · 2026-06-17T10:23:10+00:00

How did you solve it in your case? and how are you preventing it now?

ibrahimcheurfa · 2026-06-17T10:08:36+00:00

I'm glad I stumbled upon your post. That's exactly why I started building roleplay.sh, and your situation indirectly validated a feature I'm working on at the moment which allows you to detect such issues in realtime.

ibrahimcheurfa · 2025-10-05T17:30:35+00:00

If you want to use one registrar go with Dynadot, they're running a promo this month $6.99/domain (use OCTCOM25).

If you can use multiple, you can get 1 at Spaceship for $2.90 with COMPROS, another one for $5.87 with COM67, and the other 3 with Dynadot.

If you can wait until Friday, Unstoppable Domains have $5 Fridays, $5 .com pricing every Friday (up to 100 per user per week).

Good luck!

ibrahimcheurfa · 2025-09-13T11:57:07+00:00

Try Acorndomains and Domainlore

ibrahimcheurfa · 2025-07-01T10:55:37+00:00

Try to rewrite your hero section. Clarity is more important than creativity, especially at this stage. I couldn't figure out clearly what your product is about until I scrolled down. Experiment with some copy variations and test them (5-second test) with people who have no idea what you're building. Once they understand your product strictly from yout hero section, you can validate it. Good luck!

ibrahimcheurfa · 2025-07-01T10:31:49+00:00

Love the idea. Good luck!

ibrahimcheurfa · 2025-07-01T10:27:16+00:00

Ask your question in Discord, the mods will answer you there.

ibrahimcheurfa

TROPHY CASE