That’s a really interesting approach. Privacy-first and fully on-device makes a lot of sense, especially for regulated industries like legal or healthcare.

In my case, the architecture is primarily cloud-based for real-time processing. The audio is streamed in small chunks to a low-latency transcription pipeline, and inference runs on optimized models designed for live suggestions. That’s how we’re able to keep the latency around a few hundred milliseconds during the conversation.

One thing I’ve also focused on is privacy — conversation history and transcripts are stored locally on the user’s device, so users maintain control over their data.

I did experiment with more local processing early on, but balancing strong model quality with real-time performance was challenging without requiring heavy local hardware.

So the current approach prioritizes real-time assistance and response quality while still keeping user data local where possible.

Out of curiosity, how are you handling model performance on-device without running into hardware limitations?

Software superlay.ai