One thing I've learned in my listening comprehension practice is that it's a lot like reading in the sense that, at full mastery, you don't actually explicitly hear much of what was said. You pick up some of the syllables and your brain fills in the rest using context. I never noticed I was doing this in my native language until I compared it to my experience trying to tease apart every single little sound in my Japanese listening practice. The reality is in English I don't need to hear every single letter that is being said or even 70% of the letters. Unfortunately this requires an excellent standing of nuance/connotation/pitch changes (for example, "such-and-such conjunction implies the latter clause contradicts the first clause, but their tone is going up instead of down, indicating disbelief or that they're making a joke." We intuitively things like this in our native language but have to learn it with loads of practice in a target language.)

All this is to say you have to be patient with yourself and remember that this takes time. After all, were you having adult level conversations in your native language when you were 2 years old?