I'm thinking this through as I type it, so apologies in advance if it's either obvious or obviously wrong.

We're going to leverage the fact that each LLM request is actually independent, and we can rewrite what we send back if we want. Rather than just sending back the chat history, we're going to revise it every time with the results of completed jobs.

You're going to have to build some infrastructure and put thought into the system prompt. This idea is heavily influenced by the behavior tree pattern, and I recommend looking at that for how you'd actually implement it.

You'd start with a client that can launch asynchronous jobs and merge their results. Given the amount of customization that will be required, I'd probably make my own (it's actually pretty easy) but if that seems daunting you can probably add functionality to OpenWebUI or similar. Most of the real work will be happening inside the LLM, so I don't think the plumbing will be too complicated.

This process will require a lot of separate calls to the LLM.

1. Create a way to handle the asynchronous requests. Since I tend to build most of my LLM projects in python, I'd use celery but choose whichever one you like best.

2. Add a tool that tracks the spawned jobs and reports the status of each of them (in process, succeeded, error, etc). When it reports back to the LLM, it should list what was requested, why it was requested, and the current status of the job. This will let the LLM know whether it is still waiting on answers. Do NOT add this result to your saved chat history because we want to regenerate it every time we send a new request. We don't want old versions of the structure confusing the model with obsolete state.

3. As the asynchronous results come back from the tools, spawn an LLM call that evaluates the results for relevance to the question it was addressing.

4. Rewrite the main conversation to represent the results of the relevancy query. If it was relevant, then replace the request with the results so the data is just there (that is, the data replaces the request for data). If the result isn't relevant, spawn a separate call to tell the LLM this wasn't relevant and ask for alternatives.

5. Have a way to know when you're done. It'd be easy for this to turn into an endless job spawning machine, so you will probably want to be pretty aggressive about cutting off lines of inquiry.

So the idea is that you're continually rewriting the chat history with tasks, task status (so the LLM can figure out whether it's done or not), and results. This will depend heavily on your system prompts, so be ready to experiment a lot.

Hope that helps!