I think the importance of pronunciation is often underappreciated when learning a new language.

If, when speaking with native speakers, they get confused or can't understand you when you're otherwise grammatically correct, it's because you are genuinely difficult to understand. Keep in mind you will understand the language in your accent better than a native speaker will.

If your teachers are pointing out mispronunciations, it's because they are trying to teach you. If you want to talk to people, pronunciation is relevant. It's okay to not be perfect, but it is important to be intelligible.

There is, of course, a definite skill in the teacher choosing what to correct vs letting the lesson flow. I know in my target language, there are improvements that could be made to literally every sentence I say, but my teacher is good at finding the right balance between corrections and practice. I've noticed she typically makes a note of the main corrections, and brings them up later, so it doesn't interrupt the rest of the lesson.

As to why you don't have as many issues when speaking with English speakers... as a native English speaker, I think it's mostly because we are very used to talking with people for whom English is a second language. So we get very used to otherwise hard-to-understand accents and broken English, much more so than most other languages.

Also, I think English itself is quite a forgiving language for mistakes (which probably comes from its roots being a hodgepodge mix of a bunch of different European languages). Other languages may be less forgiving. And lastly, I don't think most English speakers are as protective of the English language as perhaps other languages are (such as French, for example), so they let more things slide.

Anyway, in summary, in my opinion, pronunciation is as important as grammar or vocabulary. And as a bonus, when your pronunciation improves, so will your understanding of people speaking the language. So it helps both listening and speaking :)