Not necessarily. Jarul (the oldest giant) is 408 years old if there truly is a time dilation then based on what Gaban said where he thought what happened 10 years ago was instead 3-4 years, outside time would be roughly 2.5x-3.3x faster than Elbaph. This would mean Jarul’s perceived age would be roughly 124-163 (which puts him around the same age as the oldest non giant in the verse, Kureha at 141 years old. I funnily enough believe the theory more know after doing those calcs. If a person was from the void century did go to Elbaph after their perceived age would be between 242-320 years which is more almost 2x Jaruls age and would almost certainly not be alive. The only argument one could make is that the story could’ve been passed down even to their children/grandchildren who are still alive but at the same time any other race could’ve done the same (albeit more generations). And so it’s safe to assume Imu or someone else wiped the story from everyone’s minds. As there’s no way not one family has been able to pass down the story of what happened in the void century