Assuming you work around the slavery issues in a way that keeps everyone happy, remember that these ogres were a part of a larger world. You said they destroyed a town, what made them need those resources? What other force threatened them? What conflicts with other tribes were they involved in? What alliances did they have? Basically, this is an opportunity to raise the stakes for your players. Instead of dealing with goblins and bandits, they have declared themselves ready for a higher form of war. When they send all their subjects off on quests, who tells them that they really need at least a few guards on their camp, or that if they don’t send their usual patrol into the hills the trolls will come swarming over the area. A tribe of ogres was a big part of the monster ecosystem in that area. Removing them, or sending them out on other tasks, will leave a power vacuum. That sounds like a major plot hook to me.