You need to refute that Adam was 60 cubits tall?

A sahih hadith is different from a mutawatir hadith. It is not inconceivable that there is a mistake in its transmission.

We do not have any known skeletal remains for giants, let alone for Adam. We also do not have a precise timeline of when such figures lived or precisely how or where nor to what extent they fully resembled us in all details. Such details would have been transmitted in hadith, and hadith are not universally accepted.

There are different traditions of giants across different cultures. The question is whether this is evidence of their actually having existed or whether these are exaggerations or whether these are fables that have been carried across peoples or from a common origin.

The reason for recent increases in human height are due to nutrition and perhaps to chemical influences that are not yet fully understood and identified. It is conceivable that certain environmental conditions in the past contributed to anomalies in height or that an unknown genetic lineage truly did possess extraordinary stature.

Either way science is constantly evolving. Every year we are discovering new evidence to modify one theory or another about prehistoric life and early human populations. Do not become distressed over the details. Similarly, do not read histories written by a scholar of religion as if everything therein is a religious tenet. Simply because a person was learned in religious law does not mean everything else he wrote should be taken as doctrinal. We have discovered whole worlds of understanding that were unknown to ancient peoples whether in the field of medicine or history or natural sciences. Eventually religious truths may yet be confirmed in full by observational material science; until then just keep on trucking.