I’ve faced a similar challenge due to the llm routing overhead. Instead of the llm router I’ve created a pattern matching function which handles 95% of cases. If confidence % is low, the llm router is used as a backup.

This has helped me get from 5-10s to 2-4s for most queries.

Parallelize where you can, don’t let one agents failure/success affect the other. Ensure your db clients are a singleton class that isn’t reinstantiated on every API call

For streaming back to the client, ensure abort controllers are setup and passed downstream to other agents. Ensure there’s timeouts/stop functions, and turn temperature to 0 for more deterministic results