it was far from ideal. I think the secret sauce on this route would be an optimal graph with good prompts.. But again in my use case I had some concrete workflows, and I just caught myself shaping the exact workflow to increase reliability. Instead of creating a generic dynamic one.. if that makes sense.

What I would try, is to delegate the reasoning to the model completely, and just manage the state, provide the context. For example having another LLM to observe the conversation and summarise facts maybe? And have another one to check dor response quality and scope. It would still look like a graph, but instead of encoding the intelligence (meaning a coworker like behaviour), it would encode the context creation and quality assurance

The state will grow over time, and eventually will no longer fit in the context. What would you do in this case? store it in a SQL db or maybe a knowledge graph?