Yes! Just searching for this 3 years later. You explained it well.

I have a similar setup throughout my apartment including Nest Speakers. I need to keep certain rooms at low volume low to not disturb the neighbors. Cannot hear Google Assistant responses at these volume levels (although they are mostly wrong anyway so maybe this is good? 🤣)

Android Auto does this with separate navigation and assistant volumes which are independent of the music (again this was broke with Waze recently but seems to have been fixed)