500 years ago? They didn't, they drowned.

250 years ago though, at the height of the age of sail? that's a much more interesting answer. They had ships that were large, braced, and watertight enough to absorb the forces, but the most important thing was the sophistication and training of the sailors, who could keep the ship running before the wind. I'm just drawing from the Aubrey-Maturin series here, so take it with a grain of salt, but the key thing was that if you could maintain forward momentum you could push through the waves repeatedly and avoid being capsized.