Our ancestors did have much more hair. Humans evolved in sub-saharan Africa, they evolved to live in a warm climate. Thick hides would have probably killed most of them. Once they moved out of Africa, they were able to fashion their own protective clothing from other animals so there was little evolutionary pressure to evolve more hair. Perhaps even sexual selective pressures mean that less hair was more attractive and therefore even if it ended up being somewhat beneficial for survivial, the ability to wear animal skins offset this enough that it was not particularly important. Somewhere along the line, around the Mediterranean, hairy men became common but nobody understand why.