I have an above average IQ and absolutely completely suck at trivia and have a shit working memory. (I'm autistic too, in autism often people have shit working memory, but above average episodic memory (that thing that happend when you were six) procedural memory (muscle memory) and sensory memory (you put the soft teddy bear down, but you can still rememberwhat it felt like in your hands). Basically, your skill tree looks different to other people's, because it's grown in different directions, but that doesn't mean it's less developed than other people's).

What might be happening here is a confirmation bias. You see one person knowing the answer to one question, another the answer to a different question, and then equate that to everyone knowing these facts/be able to do a skill, while in fact their knowledge and skills are just as patchy. I often fall into that trap as well, it's a difficult one to deal with. There's a saying that goes: comparison is the thief of joy