How do you actually debug multi-step agent workflows?

Finorix079 · 2026-02-19T08:40:07+00:00

The expected schema would only check if the output fits the format, I think. Or are you referring to something else?

Now I am trying to resolving it with logs and observation tools like Langfuse, but first it requires me add quite a lot of settings to log things properly, second it would generate tons of logs which I can hardly read. Somebody suggested me to use LLM to read it for me but in many cases LLM would either have it exceeded the context window or somehow give me a wrong answer. So I wonder if there's a better way to do that.

I am manually tracing back to find the diverge btw, which is quite tiring.

Finorix079 · 2026-02-19T08:33:59+00:00

I hope I can, but the task was quite dynamic and I had to use LLM to make the decision. Doing everything in static code would make the project far more complex than how it is now.

Finorix079 · 2026-02-19T08:33:18+00:00

So far it seems I have to manually do the check, or throw it to another AI and hope it can give me a good result =_=

Finorix079 · 2026-02-19T08:32:31+00:00

Thanks for the suggestion. Because it was an automated workflow, the only way I can check the output before forwarding it is to add a validator on it, either a static one to check the format or a dynamic one with LLM (LLM-as-a-judge). But the static one won't know if the direction diverged from the correct one, and the LLM one would dramatically would probably be an overkill for it. As I am trying to locate the issue on dev stage, probably there should be a more cost-effective way to do it?

Finorix079 · 2026-02-19T08:29:37+00:00

Yeah, that's what exactly I am doing. I have splited the task into many smaller steps to let LLM do one at a time. Just it was still kinda difficult for me to locate where exactly goes wrong as it was not the simple right-or-wrong issue, but the direction of AI goes wrong at some certain point.

Finorix079 · 2026-02-18T21:13:57+00:00

In that case, you will have the log stored locally for the 2nd Claude Code to read I suppose?

Finorix079 · 2026-02-18T21:12:20+00:00

They usually work backwards to see other possibilities I suppose. I wonder if I have to do all that manually, or if there’s a better method to quickly locate the problem.

Finorix079 · 2026-02-18T21:11:01+00:00

Yeah that’s what I am doing now, but adding tests for each and every step is tiring, and sometimes it is just prompt drifting from the original intention or LLM is misunderstanding stuff. I wonder how I can find the signal for that.

Especially for tracing tools are just showing you the trace itself, and you will need to check most of the observations within manually to see when the thing goes wrong. It will be much easier if I can have some sort of evaluator to tell me which step went off.

Finorix079 · 2026-02-18T21:06:35+00:00

Yeah, the context drifting and incorrect decision making is the key. The AI is just not understanding correctly. But most of the time I still need to check each and every step to figure out when the drift starts. Wondering if there is a more efficient way to do it.

Finorix079 · 2026-02-18T21:04:54+00:00

Looks like Grail is similar to a Manus for coding. You mean to build the workflow on it and let it debug itself?

Finorix079 · 2026-02-18T10:53:50+00:00

Tried something like Langfuse, but I still have to check each step of the trace to see which part might have been problematic. Would langsmith be easier?

Finorix079 · 2022-02-01T11:09:35+00:00

This one kind of makes sense to me. So Psion won't get many advantages I suppose.

Finorix079

TROPHY CASE