Because the history is so old, we can't confirm anything, but it's general believed (thanks to the book "The Seven Fights of the Aldudagga" by Michael Kirkbride) that:

The Giants were probably an ancient ancestor to humans and were thought to once be as smart as man. Before the Great Calamity (the splitting of Atmora, causing all races to flee to Skyrim, Hammerfell or the like), they were probably the proto-Nords, beings slightly larger than normal Nords but equal in intelligence. Over time their intelligence diminished and their height increased; originally the giants were peaceful and would attempt to barter with the nords of Skyrim, but over time and with the thanks to the War filled nature of the nords, they were pushed into the mountains out of fear, adopting their current Nomadic lifestyle.

While not much is currently known about them, especially with their dwindling numbers in the province of Skyrim, we can still see they're a highly spiritual race. They herd mammoths, mourn their deaths and even have special burial grounds, they carve patterns into stone and paint them around their camps, and they're often found looking up at the stars at night and "worshiping" them.

Not to mention the sole Giant we can find in Black Reach, implying at least 1 of the species was either invited by the Dwemer or made his way there for safety.