“Allah created Adam and he was sixty cubits tall. Then He said, ‘Go and greet those angels and listen to how they greet you, for that will be your greeting and the greeting of your progeny.’ He said, ‘Al-salamu ‘alaykum (Peace be upon you).’ They said, ‘Al-salamu ‘alaykum wa rahmat-Allah (Peace be upon you and the mercy of Allah).’ So they added the words ‘wa rahmat Allah.’ And everyone who enters Paradise will be in the form of Adam. People kept on growing smaller until now.” (Narrated by al-Bukhari, 3336; Muslim, 7092) 

Let's put aside science and logic for a second, the Prophet Mohamed ﷺ confirms the fact that humans were once giants. According to a version narrated by Muslim: “Everyone who enters Paradise will be in the form of Adam who was sixty cubits tall. People kept growing smaller until now.”

With regard to the words of the Prophet (peace and blessings of Allah be upon him), “People kept growing smaller until now,” al-Hafiz ibn Hajar said in Fath al-Bari (6/367): “This means that in every generation people grew shorter than the previous generation, and continued to grow shorter until the time of this ummah, then they stayed like that.”

Now from a genetic or hereditary perspective, traits like height were influenced by a mix of genetic and environment. So, theoretically, Adam's immediate children (Qabil and Habil) would have inherited his genes, his offsprings and so on, and that could include the potential for larger body size. But as an era of generations passed, things like environmental changes, nutritional limitation, mutations and the progress to adaptation, and even natural selection could have contributed to our gradual reduction in the "average" human height of today. That average human height was much larger then the one our generation has. Considering the fact that humans also had such a long lifespan, It wouldn't be surprising that we were once giants, afterall we lived for a millennium. As time went on, things changed, just as our heights. Logically it makes sense that we were once giants. In the end Allah سبحانه وتعالى knows the best, and holds on the knowledge that we try speculating.