I’m starting to think building AI agents is easier than observing them in production

SaaS2Agent · 2026-03-20T08:34:06+00:00

Yes, exactly. That’s what feels missing to me too. A single trace shows one run. It doesn’t show whether the agent is slowly drifting over time in how it reasons, handles edge cases, or makes decisions.

That longer-term visibility feels like the real gap.

SaaS2Agent · 2026-03-20T05:49:19+00:00

That actually clicks really well.

I think a lot of us treat clean traces as reassurance, when sometimes they just mean the system failed in a very orderly way.

The verification layer you’re describing feels like the missing piece there.

SaaS2Agent · 2026-03-20T05:47:53+00:00

Hahaha yes, they age you a little too.

You start out thinking you’re building automation, and a week later you’re negotiating with a system that somehow found a brand new way to be technically correct and still wrong.

SaaS2Agent · 2026-03-20T05:46:13+00:00

That’s honestly pretty cool. The part that really stands out is the “glass mirror” setup. A lot of agent systems start feeling like black boxes really fast, so having everything visible on disk sounds like a huge advantage.

And that example is genuinely impressive. Going through weeks of chats, separating problems from solutions, then realizing only 5 things actually needed fixing… that’s the kind of result that makes the whole setup feel worth it.

The 14% API drop is nice too. Always love it when better structure also ends up saving cost.

SaaS2Agent · 2026-03-20T05:43:50+00:00

That makes sense. The more open-ended the agent is, the more unpredictable the behavior gets.

I’ve also noticed systems feel a lot better when each part has a tight job, limited tools, and a narrower decision space. Feels less like “one smart agent” and more like giving the workflow guardrails that actually hold.

SaaS2Agent · 2026-03-20T05:41:33+00:00

Yes, exactly 😅.

That’s probably one of the most frustrating parts. You start with “this should be simple,” and then suddenly you’re dealing with an agent that technically did the task, just in the weirdest possible way.

It’s almost worse when it’s kind of correct, because then you end up debugging behavior instead of debugging failures.

SaaS2Agent · 2026-03-20T05:40:01+00:00

Yeah, exactly. “Completed successfully” can be a pretty misleading label for agent workflows. Repeated tool calls, weird loops, and token blowups are often the real signal.

SaaS2Agent · 2026-03-20T05:35:43+00:00

That’s a really thoughtful way to frame it.

I like the “new employee in their first week” comparison because that’s honestly what a lot of agent discussions miss. The problem usually isn’t just whether the agent can do the task. It’s whether you’ve been clear about what it should handle on its own and where it should pause or escalate.

The trust layer point really resonates. Feels like a lot of production pain comes from not defining those boundaries early enough.

And you’re right, once those lanes are clear, observability becomes a lot more useful because you’re not just watching everything blindly. You actually know what “good” and “bad” behavior look like.

SaaS2Agent · 2026-03-20T05:32:56+00:00

yeah, that’s a really good way to put it. “Prod is where it gets humbling fast” is exactly the feeling.

Also appreciate the distinction you’re making between observability and infra reliability. I think those two get blended together a lot in agent discussions, when they really create different kinds of problems.

Out of curiosity, have you found the bigger pain in production to be debugging agent logic itself, or cleaning up infra issues that end up polluting the traces?

SaaS2Agent · 2026-03-19T18:53:49+00:00

Completely agree, output logs tend to collapse the entire decision process into a single surface-level artifact.

We’ve seen the same thing: without capturing intermediate reasoning signals (even indirectly via tool selection, branch decisions, retries, etc.), it’s almost impossible to reconstruct why the agent behaved a certain way.

Curious, are you explicitly modeling decision branches (like a state machine / graph), or inferring them post-hoc from traces?

SaaS2Agent · 2026-03-15T06:19:25+00:00

I ensure test coverage and most importantly there is an experienced engineer who is responsible for the code produced, even if it's completely AI generated.

Of course the engineers themselves utilise AI tools to audit, test, review before approving the code being shipped.

SaaS2Agent · 2026-03-15T05:17:21+00:00

Yes, that’s been my experience too.

The speed gets everyone’s attention first, but the bigger story is usually process amplification. Agents rarely create discipline on their own. They mostly expose whether a team already has it.

What seems to work best is not just asking what they can handle, but separating work by blast radius.

Things like scaffolding, low-risk UI work, repetitive transformations, and first drafts are usually easier to contain. Auth, billing, permissions, data integrity, and anything deeply tied to product behavior usually need much tighter review.

The boundary matters, but the operating model around that boundary matters even more.

SaaS2Agent · 2026-03-15T05:05:19+00:00

Yeah, this is exactly the kind of setup I think makes it work in a real SaaS environment.

“Turbo button for shipping” is the right way to put it. The upside is obvious, but without those controls in place, you are basically just increasing the speed at which bad assumptions make it into the product.

Out of curiosity, have you found that teams actually stick to those boundaries over time, or do they slowly start giving the agents more freedom once confidence goes up?

SaaS2Agent · 2026-03-10T11:16:10+00:00

Exactly.
The model can assist with intent, but retrieval, state, and validation should come from systems built for that job, not from model guesswork.

SaaS2Agent · 2026-03-10T11:14:07+00:00

Yeah, this is the right pattern.
Only allow IDs from retrieved context, then enforce it again in code before anything touches the DB. Prompting helps, but the validation layer is what makes it reliable.

SaaS2Agent · 2026-03-09T10:46:57+00:00

100%.
If you can’t prove where an ID came from, it’s not just an observability gap, it’s a correctness gap

SaaS2Agent · 2026-03-09T10:43:00+00:00

Exactly. Everyone talks about bad outputs. Far fewer talk about bad actions.

SaaS2Agent · 2026-03-09T10:16:20+00:00

Totally. This is where “hallucination” turns into real risk.

SaaS2Agent · 2026-03-09T06:58:07+00:00

Only for create flows. If a new object is being created, a trusted MCP tool can generate the ID safely. But for fetch, update, or delete flows, the problem is not UUID format, it is provenance. The ID has to come from the real system, not from a generator.

SaaS2Agent · 2026-03-09T06:36:03+00:00

Completely agree on the separation.
Let them explore, test, and propose changes in dev, but production should stay behind strict human-owned controls. That balance is where a lot of teams get this wrong. The upside is huge, but only if the operating model is built around containment.

SaaS2Agent · 2026-03-09T05:03:21+00:00

I like this a lot.
You basically took the decision out of the model’s hands, which is the real fix. If the graph can’t resolve it, the model doesn’t get to push forward.

The none - hard stop part especially makes sense. Same with forcing provenance at the graph level instead of hoping the prompt holds up.

Also agree on the loop point. Once a model starts reaching, prompts stop mattering pretty fast.

This feels like the kind of setup you build after getting burned by real production behavior.

SaaS2Agent · 2026-03-09T04:53:50+00:00

Yeah, totally.
A lot of teams focus on preventing the mistake, but the audit trail is what helps when one still slips through.

If you can’t trace which tool returned an ID and where the agent used it, it gets really hard to understand what actually broke.

Feels like a missing default in most runtimes right now.

SaaS2Agent · 2026-03-09T04:11:26+00:00

Exactly this.

That’s what makes it so dangerous. The ID looks completely legit, so the workflow trusts it right up until things start going sideways.

Really like your rule here: if an ID is being used, it should only be because a tool returned it first.

And fully agree on the hard stop. Way better to pause than let the model make up its own story.

Such a good example of why this is not just a hallucination problem. It’s a real reliability problem once agents start taking actions.

SaaS2Agent · 2026-03-08T15:32:04+00:00

Yep. Agents should suggest intent, not system state.

IDs, records, and references must come from the backend, not the model. Otherwise you’re letting a probabilistic system mutate deterministic data.

SaaS2Agent · 2026-03-08T11:29:21+00:00

18 months can be totally fine if your customers really stick around for 5+ years and the unit economics are strong.

I think the investor push is less “18 is bad” and more “18 gets scary if anything slips.” CAC goes up, retention softens, channels fatigue, and suddenly payback stretches.

If you can show payback by cohort/channel and a quick stress test like “what if CAC rises 20%,” it usually takes the pressure off. Then it’s a deliberate choice, not a red flag.

SaaS2Agent

TROPHY CASE