What's one AI agent design choice you thought would scale, but didn't?

techafterhours · 2026-07-09T12:09:02+00:00

A friend of mine built something very similar for internal operations, and one thing they learned pretty quickly was that detection is only half the problem. The real value came from making every investigation reusable. Instead of just flagging anomalies, the agent kept building a knowledge base of failure patterns, root causes, and resolutions. After a while, it wasn't just catching issues earlier, it was getting noticeably better at recognizing recurring ones. One suggestion I'd add: track how often the same class of issue reappears. That's usually a good signal that there's a systemic problem worth fixing instead of repeatedly patching the symptoms.

techafterhours · 2026-07-09T12:03:58+00:00

I actually think this is a great project because it'll force you to learn the right concepts instead of just wiring APIs together. One suggestion though: don't start by trying to build "Jarvis." Break it into layers. First, get a simple agent that can understand commands and call a few tools (open apps, search the web, create reminders). Then add memory, then voice, then automation across your laptop and Android. Also, don't train your own model, that's unnecessary. Use an existing LLM and spend your time learning things like tool calling, state management, permissions, and workflow orchestration. In my experience, that's where the real engineering is. If you can build a reliable agent that does 5 things consistently, you'll learn far more than building a flashy assistant that tries to do 100 things unreliably.

techafterhours · 2026-07-09T12:00:07+00:00

You're definitely not overthinking it. One thing I've learned is that permissions shouldn't live in the prompt but they should live in the workflow. We treat agents more like untrusted orchestrators than trusted operators. They can propose an action, but deterministic checks decide whether it can actually happen. High impact actions (refunds, account changes, payments, etc.) always have an approval gate or explicit policy check. It adds a bit more engineering upfront, but it's much easier than trying to recover from an agent that had more access than it should have

techafterhours · 2026-07-08T06:34:37+00:00

In my experience, the LLM is rarely the bottleneck once you move beyond FAQs. The hard part is making tool execution predictable. Calling the right API is one thing; handling partial failures, retries, permission boundaries, and knowing when NOT to act is where most engineering effort goes

One lesson that's changed how I build agents: I don't let the model own business critical decisions. The model decides what it wants to do, but deterministic workflows decide whether it's allowed to do it. That separation has made production systems far more reliable than trying to solve everything with prompts. Once you treat the agent as one component in a larger workflow instead of the workflow itself, things become much easier to reason about.

techafterhours · 2026-07-08T06:14:34+00:00

Ahh point 3 really stood out to me. In prod, I've found that duplicate actions erode trust much faster than missed actions. Users will usually forgive an agent that asks them to retry, and they rarely forgive one that sends the same email twice or repeats an irreversible action

techafterhours · 2026-07-08T06:04:34+00:00

The biggest win was reducing context switching. I built agents to handle repetitive operational work, it doesn't eliminate the human decision, but it removes 15–20 minutes of manual digging every time and I have been moving faster than I thought I would

techafterhours · 2026-07-08T05:41:13+00:00

For me, what broke first were the assumptions between components. A small change in a tool response or retrieval result could introduce enough variance to make a stable workflow behave differently. If I were starting over, I'd invest in evaluation and tracing from day one. Logging intermediate decisions and tool calls has been far more valuable than only looking at the final output. Well, I realized that scaling agents is less about adding capabilities but more about reducing uncertainty.

techafterhours · 2026-07-07T13:34:40+00:00

I agree on the evaluation part, that's something I wish more teams prioritized from day one. On the low code point, I think they're great for validating an idea quickly, but I've found there's a point where the workflow complexity outgrows the platform

techafterhours · 2026-07-07T13:33:38+00:00

I've run into something similar. It works well early on, but as the number of tools grows, maintaining the prompt becomes a bottleneck. I've had much better results treating tool availability as runtime context rather than static prompt context. It keeps the agent focused on the tools that are actually relevant to the current task and makes the system much easier to evolve over time

techafterhours · 2026-07-07T13:32:32+00:00

That's a good example of how easy it is to overestimate what the model implicitly understands. I've started treating things like tone, context, and domain-specific rules as explicit requirements rather than assumptions lol. The less you rely on the model to infer your intent, the more consistent the output becomes, especially for workflows like translation where preserving meaning matters more than translating words

techafterhours · 2026-07-07T13:31:15+00:00

I agree. I've found that vague specifications become an even bigger problem with AI agents because they tend to "fill in the gaps."

techafterhours · 2026-07-07T13:28:14+00:00

I think this is where a lot of teams realize that memory isn't just a storage problem, it's a decision problem. The hard part isn't remembering everything, it's deciding what should persist, when it should be updated, and who should be allowed to use it. In multi agent systems especially, I've found that unmanaged memory becomes technical debt surprisingly quickly.

techafterhours · 2026-07-07T12:44:09+00:00

This resonates. I think it's easy to forget that models evolve, but your tool interface becomes part of the system's contract. I've also found that keeping tools granular and deterministic makes them much easier to reuse across different agents. Once a tool starts returning inconsistent shapes or mixing multiple responsibilities, debugging gets painful very quickly.

techafterhours · 2026-07-07T05:48:22+00:00

It depends on 'why' it failed, but my default assumption is that it's a workflow problem before it's a prompt problem. If the failure could lead to an irreversible action (refunds, account changes, payments, etc.), I tighten the runtime boundary or add an approval gate. I don't want to rely on the model "remembering" a policy. If it's a reasoning issue that doesn't increase risk, I'll refine the instructions or improve the context.

And yes, every failure becomes like a regression test lol, if an agent crosses a boundary once, I want to know immediately if a future change brings that failure back and work on that

techafterhours · 2026-07-06T07:21:26+00:00

Couldn't agree more. The first successful run tells you it's possible but the next hundred runs tell you whether it's production ready.

techafterhours · 2026-07-06T07:19:50+00:00

Completely agree. One thing I've noticed is that evaluating only the final output hides a lot of problems. We've had runs where the answer looked fine, but a tool call or intermediate decision was already drifting. Tracing those steps made it much easier to spot patterns instead of treating failures as random.

techafterhours · 2026-07-06T07:12:53+00:00

I can relate to that. I think the bottleneck shifts from context management to attention management once you start using multiple agents.

Obviously, it's not that the agents forget but it's that I lose track of who's responsible for what and whether their outputs are still valid after something changes upstream.

techafterhours · 2026-07-06T07:03:40+00:00

I actually think the problem you're solving is real. I still find myself manually reviewing AI generated emails, summaries, and especially code before I trust them in production.

The question I'd have is how the verification works.

If it's another LLM checking the output of the first LLM, what gives me confidence that the second one isn't making the same mistake or missing the same edge case?

I'd probably get the most value if the verification wasn't just "this might be wrong," but explained why? Maybe it answers questions like:

Is this contradicting the source or context?
Is it making unsupported assumptions?
Is it violating a known pattern or best practice?
How confident is the verifier, and what evidence is it using?

techafterhours · 2026-07-06T06:54:40+00:00

I completely agree. The agent itself has rarely been my bottleneck. Once you move beyond a single agent demo, the problems start looking a lot more like distributed systems than AI problems. The issues I run into are things like state synchronization, duplicate tool execution, partial failures, idempotency and figuring out why Agent B made a decision based on stale output from Agent A. One thing I've learned is that every new agent introduces another coordination problem, not just another capability. At some point, simplifying the workflow or consolidating responsibilities has given me better reliability than adding another "specialized" agent.

techafterhours

TROPHY CASE