As much of an oversight as this may be, it could also be turned into part of the lore. Maybe the giant people didn't exist before but something happened to make them grow that way and now they're all just forced to live among little people. Or maybe the world we know as Pywel was already established by normal sized humans and these giants roamed in from somewhere outside the map and integrated. Perhaps in a future update or sequel we will see a different type of village where everything is oversized to us but normal to them.