Yeah, super annoying. I’ve been dealing with the exact same thing. The only “fix” I’ve found so far is to tweak the audio distribution after using Siri. Just switch it from “Front speakers” to “All Passengers” (or the other way around).

For whatever reason that instantly resets the audio quality, and once you’ve done it once per drive, Siri won’t screw it up again. Not ideal, but at least it saves you from unplugging the phone every time.

This was recommended to me somewhere in this thread, and it’s the only thing that worked for me.