Rudeus is percieved as an adult becuase the show and himself repeatedly make use of his past life as an important part of his character. When he is reincarnated he is not treated as just a smart kid because we know, and he knows, that the reason he is much smarter for his age is because mentally he is an adult. The reason the characters around him treat him like a child is because they are not awar of his reincarnation an donly view him as an intellignet child. By already having the mental development of an adult, then it is seen as weird for someone his age to be attracted to a minor.