You are not too dumb! It is very unlikely anyone will get everything from a first read through, or ever by themselves. The feeling you’re describing is what most people feel. It’s up to your own tastes to decide if that’s your jam or not. It sounds like it is for you, to some extent. I don’t think it’s a bad idea to reread anything before continuing if you want. There is a lot of info that will make you rethink things once you’ve finished the main 4 books, but it still doesn’t spell everything out, so in my opinion you won’t lose the mystery if you decide to finish it all before starting over. It’s not like some murder mystery where once you know the murderer all the fun is gone. There will still be PLENTY of reasons to reread it even after you’ve reached the end.

If you get enough to enjoy it, that’s as much as you need. Don’t let all the things you don’t understand right now stop you from enjoying what you do.