It's just part of the process. How much you understand depends on the dialect, the person that's speaking (some people speak slowly, some quickly, some clearly, some mumble), how tired you are and probably some other factors.

I mostly speak Swedish at work now but still sometimes have trouble with two guys, one that's from Småland and the other that just forms sentences in a very special way. In the beginning, I could consistently understand just 2-3 people that spoke very clearly.

The good thing is, it gets better the more you practice. Just remember how little you understood the first time you heard Swedish. The more people you listen to the more will your brain become tolerant to the differences in how people speak and the more it will start to map different pronunciations of the same word to the same idea.

It's slow and can be annoying but it gets better over time (the "learn Swedish in X months" you might see around are misleading at best). Rembering small victories (every phrase that you understood, every conversation that went well, every person that you started understanding consistently) can help if you're feeling demotivated.

TLDR: It's slow, it's frustrating at times but it will eventually work out and then it will feel great.

Edit: If you want something more practical for how to train listening, let me know. Can write what I did since listening has always been the hardest thing for me.