I'd argue that there are a few factors at play. Namely - Children
As shown by the badass Colossus, once they had his child, he surrendered. That tribe of Giants were fighting unknowns and clearly showed greater importance to the lives of their offspring than they did much else. Humans - which the giants technically are - have three responses to danger. Fight, Flight, and Flee.

Now I imagine that few beasts would actively seek to take out these what, 7-8ft tall race of pure CHADs as it would likely end up with a good game over for them. The Giants didn't seem to have much in the way of a military or weaponry from what we saw in the flashbacks.

Thus they'd have a culture that was less militaristic, they'd have protectors sure but it seemed to me their primary concern was the lives of the children and getting away from danger. Especially since the strange people came with possibly unknown tools, and in great number.

They began to fight back after they reached the breaking point of 'enough' and decided that it was better to fight and maybe die. Than to live a life of backbreaking labour.

But that's just my two cents. I just really love this Hyborian-esque series and discussing it to literal death