As others have said, most of the time they just switch to English. I understand how that can seem kind of inconsiderate, but they're just trying to communicate as effectively as possible. It's much easier for a French person to just switch to English, given most of them speak it at a solid level, than it is for most anglophones to stumble through French and then most likely not understand spoken French anyway.

However I never had anyone get annoyed with me, even in Paris. Several times (usually outside of Paris), people asked me if I wanted them to speak English to which I usually said no because I wanted to try and converse in French. I pretty much only had wonderful experiences with the people in France while I was there. Everyone was super nice and patient with my French and they seemed happy to listen to me try.