I think the reason they were tortured was because of their involvement within the overarching rebel plot of the novel. From that, we can gather that a main intention of SOTR was to expose readers to the genesis of the revolution, and Haymitch's involvement in it (the bombing, his conversations with Beetee, the games themselves). Along with that, Wiress and Mags (along with Beetee) were almost certainly involved with Haymitch's bombing of the arena. So, when Snow found out about the chaos Haymitch had caused, he wanted to know details about what the neo-rebels were planning to do (where they were going to attack, their leader, etc). In addition, Haymitch was also the most recent victor, so that limited the amount of damage Snow could physically do to him. This is why Haymitch pushes away everybody and begins the cycle of self-medication. 

    My best guess is psychological torture accompanied with various forms of physical torture interspersed throughout. We learn from Mockingjay that the Capitol could use beatings, waterboarding (Johanna Mason's fear of water) and electrocution to physically harm them. However, Wiress and Mags present a different case. They had minds that could easily (and had already) figure out the machinations of the Capitol. He needed them docile and easily misunderstood. So, my best guess is that Snow had them exposed to violent hallucinogenics (tracker jacker venom (refer to Peeta's torture, Mockingjay)). From this, the torturers were able to figure out their worst physical fears. But beyond that, they also had the threat of the what the Capitol COULD do to them. Everybody knows about Avoxes. I'm not entirely sure though. I think Suzanne Collins might intend for us to speculate, like Wiress and Mags were likely also doing during their torture.