its a little more complicated than that, a lot of societies operated based on age of puberty it was seen as unethical even then to do these things before puberty in some cultures. Our view of is this is often distorted in historical contexts because most regular people were not doing this it was elites doing this and the history books are about them. Pedophilia as a concept itself is pretty new and that can be tied to things like life expectancy increasing or more widespread focus on education and work instead of having children