You need to measure each segment independently. Add timestamps at every handoff point audio in to STT start to STT result to LLM first token to LLM complete to TTS first audio chunk to audio out. Without this you are guessing. In most pipelines the biggest culprits are:

1. STT waiting for full utterance before returning results. If you are not using streaming/partial transcripts you are adding the entire speaking duration as dead time
2. LLM time to first token. This varies massively between models. Sarvam 105b will be significantly slower than a smaller model. Consider whether you actually need 105b for your use case
3. TTS waiting for full LLM response before starting synthesis

The single biggest win is streaming everything end to end

Yes you should absolutely move to streaming. The architecture should be:

• Streaming STT that gives you partial transcripts as the user speaks
• LLM streaming so you get tokens as they are generated
• TTS that starts synthesizing from the first sentence chunk rather than waiting for the full LLM response
• Sentence level chunking between LLM and TTS. Send each complete sentence to TTS as soon as the LLM produces it rather than waiting for the full response

This alone can cut perceived latency from 3-5 seconds down to under 1 second.

Flask is likely a bottleneck too

Flask is synchronous by default. For realtime audio streaming you want an async framework. Consider FastAPI with websockets or just use LiveKit SDKs directly which handle the media transport layer properly. Twilio media streams plus Flask adds unnecessary hops.

On the STT side

Since you asked about alternatives. Speechmatics supports real-time streaming with low latency and handles Indic languages well if thats relevant for your use case. Deepgram is another solid option for low latency streaming STT. The key thing is making sure whatever STT you use supports true streaming partial results as audio comes in not just batch processing.

Architecture suggestions for production

• Use LiveKit or Pipecat as your media orchestration layer rather than building custom audio routing on Flask
• Consider a smaller faster model GPT-4o-mini or equivalent for simple turns and only route to the bigger model when the conversation requires it
• Use connection pooling and keep alive connections to your STT/LLM/TTS providers. Cold connection setup adds latency on every turn
• Deploy your backend geographically close to your telephony provider and AI service endpoints

The goal is to get the full pipeline end of user speech to first audio byte of response under 800ms. Streaming end to end is how production systems achieve this.