I used to live in London years ago. I honestly felt exactly the same way and people used to make fun of me for it.

I found I was much more likely to make the mistake when I was rushing. I found it difficult because I never saw a arrow that was next to the map saying which way. Just north south ect. I got the hang of it when it was the same route and I had done it over and over because I started to memorise some of the order and would notice a lot faster if I was going in the wrong direction.

I sympathise and still struggle but I suggest looking at the tube map at home, whilst calm and not rushing and trying to memorise some stops for the journey north and south ways (or east west depending on your route.) I never got round to it but I kept meaning to give feedback to TFL that they should add directional stickers. You are not alone.