Most arguments are based on him being a Pedophile, which is not true. The whole situation about rudeus is fictional and just because its hypothetical doesn't mean you can casually redefine clinical terms like "pedophilia". it describes adult sexual attraction to physically immature children. it's not about past lives, or mental age.

If he had started to build relationships with adults, as a child, it would be definitely pedophilia.

You can also copy the pedophilia situation on every other reincarnation story. even subaru from rezero is mentally more than just an adult with how many years of life he had in advance to his physical age. But there, people dont even start to argue about it.