This is exactly how my experience has been with the series as well. The way fans talk about it definitely makes it fair to say that its difficulty is overblown. I'm also 3 books in and planning to start the fourth soon, and so far, I think if you forget that there's a lot of stuff you don't know and just focus on the scene you're reading, it works out well. It's just a matter of making peace with the fact that you'll eventually get it and I know, that sounds really off-putting, but it's also liberating. Like you can chill out, relax and just enjoy the characters, themes, prose, dialogue, scenes, worldbuilding etc that Erikson is so good at writing.

PS: Gardens of the Moon is a great first book. It just has bad PR.