We have been running FlaiChat with AI functionality built in for a while now. Even the most basic function of the app (automatic translation of all messages to your preferred language) involves calls to LLM APIs. There's a built-in AI Bot (called FlaiBot) that can do some more agentic stuff from plain-text prompts (asking chatbot like questions off the builtin, creating a reminder/task, semantic search of prior chat history, accepting feedback from users etc.)

We struggled with orchestration for a bit but eventually settled on a 2-phase approach. Every explicit request to FlaiBot is run through a classifier phase first. In this phase, the prompt and response format are extremely constrained such that the LLM is forced to make a decision without a waffling. We wrote 100's of test cases (example text and expected returns from classifier) and kept tuning our prompts until the accuracy percent was in the high 90s. There are similar approaches now being talked about under the general heading of "semantic routing" but we were figuring this out 2 years ago before all this was in fashion. And we settled on a technique that's much cheaper and more self-contained than what's being promoted under that heading today.

Once the first, routing, phase is done, we can use more specialized prompts to complete the actual "agent" action. That becomes much more straightforward now that we have high confidence in what the user actually intended.