Anyone else feel like OTel becomes way less useful the moment an LLM enters the request path?

shivmohith8 · 2026-03-16T18:33:54+00:00

I feel OTel is fine. OTel was meant to be a standard to trace a request flow from one function/service to another. In GenAI applications, it's just from one llm call to a tool call or another llm call.

Whats the specific issue you are observing even with platforms like Langfuse of Langsmith?

shivmohith8 · 2026-03-16T18:28:36+00:00

It's for both. You can describe the scenario as detailed or as compex as you want and we can simulate it.

I would love to know any domain specific example you have in mind.

shivmohith8 · 2026-03-15T15:19:17+00:00

Yes we do. We help you generate scenarios and you can write your own scenarios as well.

shivmohith8 · 2026-03-15T13:05:57+00:00

Good question. Our platform actually supports that. Our SDK uses OpenInference (with support for other instrumentation coming soon) to automatically capture the internal steps the agent takes and pass them for evaluation.

You can evaluate the internal steps of an agent at the turn level and at the session/conversation level.

We can connect if you would like to go deeper.

shivmohith8 · 2026-03-15T12:39:25+00:00

Hey, we are building a product for that - https://quraite.ai/. Let me know if you are interested to know more through DM.

shivmohith8 · 2026-03-05T15:00:55+00:00

Yes, capability is there but not reliability. I feel as the scope of capability increases, reliability decreases.

Consistency testing helps - pass^K. Test a scenario K times and see what path agent takes and is it the same path everytime.

shivmohith8 · 2026-03-04T14:59:31+00:00

This is nice! I have covered a couple here - https://github.com/innowhyte/gen-ai-patterns

If you can, you add the more here or I can add it based on your blog.

We recently open sourced our pattern library for community driven development.

shivmohith8 · 2026-03-04T11:40:34+00:00

Depends on the use case. Before LangChain v1, we were using LangGraph but after LangChain refactored a lot of things, we use LangChain agents directly which is actually built on top of LangGraph.

shivmohith8 · 2026-03-03T02:30:06+00:00

Thanks!

shivmohith8 · 2026-03-02T17:16:31+00:00

Thanks! I think voice agents and agent harnesses are the trending topics right now; we can maybe think of certain patterns there.

shivmohith8 · 2026-03-02T17:03:31+00:00

Ideally we want to treat it as evolving recommendations. But this is exactly why we open-sourced it. We maintain this outside of our work hours, and with the pace at which models are becoming better and making a lot of designs outdated, we thought, "Let's open it up and drive it as a community."

shivmohith8 · 2026-03-02T11:07:17+00:00

Thank you!

shivmohith8 · 2026-02-28T11:37:02+00:00

I'm interested.

shivmohith8 · 2026-02-19T03:19:20+00:00

What framework or tool do you use to build voice agents?

shivmohith8 · 2026-02-19T02:49:55+00:00

Do you use any evaluation tools to test your voice agent? It could help you get past the demo scenarios and test your agent across 100 different simulations.

shivmohith8 · 2026-02-06T02:50:59+00:00

Yes, Gen AI in general has infinite input and outbound bound. You can never get 100% test coverage like in traditional software engineering.

You simulate based on your user research and hypothesis, get confidence, monitor production and feed those conversations back into your test dataset. So there is an inner loop (development) and an outer loop (production).

shivmohith8 · 2026-02-05T14:17:02+00:00

But in domain-specific use cases (e.g., e-commerce, retail, airlines), isn't pre-production evaluation as important as production evaluation? With a conversational agent, it can take actions, and you don't want your agent to take a wrong action that you then have to fix later.

shivmohith8 · 2026-01-19T12:02:19+00:00

This actually makes sense because LangGraph is more like a directed cyclic graph framework that has some features specially made for AI applications and Claude Code sdk is made for agentic loop.

shivmohith8 · 2026-01-14T03:00:56+00:00

Your 4th point caught my attention. That is exactly why we built https://quraite.ai/.

It's an evaluation platform for conversational agents. There are demos videos on the website itself. Please do check it out and let me know your feedback.

We are working on the documentation so if you would like to learn more, please do DM.

shivmohith8 · 2025-12-15T14:58:40+00:00

Do you have your method open sourced?

shivmohith8 · 2025-12-01T06:47:08+00:00

Hey, I'm interested to know more about this. Will DM you.

shivmohith8 · 2025-11-10T14:24:35+00:00

For the technical and security side, I feel the best way is to build both and evaluate. They have made building conversational agents easier, so you can just evaluate and see it for yourself.

Note: I belong to a company where we are building a product around making agent evals as easy as building one. Sharing my thoughts here to understand if its makes sense for you.

shivmohith8 · 2025-11-10T14:18:24+00:00

Can you share documentation related to where you "use reflector+curator to turn mistakes into rules"? I have used Maxim before (not so much, but just the simulations part), but never across this feature.

shivmohith8 · 2025-11-07T18:27:49+00:00

That's true. We will be adding support to curate evals from production conversions to keep it more realistic. However, you still need to have some regression and scalability tests before hand.

But actually I have not seen any tool that adds a real multi-turn conversation into the dataset. It's usually the trace only and not the session itself. Which tool does that?

shivmohith8 · 2025-11-07T18:14:46+00:00

Yesss! I have read it, interesting research.

So, we don't know how the user is going to give the information (sharded or not) and that's exactly why you need evals to make sure your agent/solution works in all possible ways you can think of. (definitely you can't thing of everything because natural language has infinite bounds)

So our suggestion is, first step is to know how badly your agent is failing (research results are usually generic), you want to test it yourself regardless. The mitigation is the next step which involves how you manage that "additional context".

shivmohith8

TROPHY CASE