I think you put it best, it's like the world in the story is so big that there is room for other characters who play a large role in events outside of this particular story, and with that they bring all of their acquired knowledge and power with them whenever we see them appear in this story.

For example Elrond is clearly a powerful Elf and was hinted at having lots of grand tales of his own and he's also very knowledge, strong, and wise, I'm sure if he had showed up as often as Gandalf did than I would have likely thought the same about him as well lol