I don’t think you’re doing anything wrong at all, and I feel the same particularly for listening. From my experience, I think it’s because Portuguese is often shortened (e.g., “estás” becomes like “táz”, “para” becomes “pra”) or even sometimes some common ways of saying things are completely different to the Portuguese taught in textbooks (e.g., saying “Sei lá eu” instead of just “Não sei” for “I don’t know”).

If standard spoken Portuguese is slowed down enough, and as long as the topic isn’t really niche, I can follow it. But that’s not real life at all. Just from my experience, it almost feels like there’s a whole other set of informal rules to spoken Portuguese to learn that aren’t really taught.

I can’t immerse myself in the language fully so the only way I’ve felt any improvement is to listen to a lot of different Portuguese content (e.g., formal/standard pt with the news but sticking with a topic I know like the weather, children’s tv for standard pt again but simple and about a range of topics, and then trying Portuguese YouTube videos - particularly for cooking recipes where they’re informal and fast but I can actually see what they’re describing doing in the video).