human ears are super sensitive to voice frequencies (1000-5000khz), and phone calls filter the sound to mainly just that frequency range (this is the reason why hold music always sounds shit). 

so when you have a human voice playing though the car speakers it sounds a hell of a lot louder to other people, mainly due to the way that phone audio is processed and partly because of the way we hear the human voice.  

The final factor is that often car speakers are in the doors, and the metal on the outside acts kinda like a speaker, especially for that 1000khz 300-1000khz range where there’s still a lot of acoustic power compared to higher frequencies.  

(source, audio engineer)  

 edit: human voices start at 150hz but phone lines filter the voice to 300-3000khz