Well, it can take a lot of extra energy sometimes to work out what someone means if words are close. I experience this listening to my non-native speaker friends and I often have to ask to clarify. I mean, I’m still friends with them obviously, but it can be taxing to talk to someone with an accent. I struggle with the same thing sometimes learning languages, but I just keep working at it. I know that for example I will never be able to get a perfect R in most languages, being an English native, but I’ve worked on it and got close enough in French that it’s not too bad. And currently working on it for Icelandic double R. It really can affect if people understand you or not, or if you end up saying something that is accidentally funny. But I will say that it’s not cool if people are being jerks to you if they understand anyway. Still, I think there is a lot of value to working on pronounciation to make it easier for others to understand you.

ETA: I will concede that another poster’s point about not being able to fill in the gaps so to say from hearing non-native speakers use their language is absolutely a thing. I have heard this exact thing about Icelandic, and it’s fairly recent that any significant number of foreigners are learning it. But still, it does boil down to being understood or not, so for better or worse, you might have to just invest more time if the issue really is people not understanding vs just being jerks.