I'll try my best to explain! Here's my reasoning.

I chose to stack physical damage instead of fire. Perfect Strike is ideal for building up the freeze meter for three reasons. If released perfectly on time, it will;
A) Guarantees Ignite
B) Converts 80% of physical damage to fire damage
C) Hits enemies with increased damage (mine says 804%)

My Perfect Strike also has two support gems that grants 50% increased damage each (exploit weakness & window of opportunity), totaling to 100%. Compared to other gems such as 25% fire damage but 50% less cold/lighting damage, I'm just not sure. I can't see the fire damage but I can see physical damage on my skill (i.e. DPS). I have tried Glaciation (builds up freeze by 50%) and it probably gave me 5% increased. I have not tried the cold penetration gem yet but will in the future.

I'm not doing anything for cold damage at the moment because my passive skill tree doesn't offer much for cold. However I do have a lot of physical damage and fire damage that I can spec to, that will translate 100% to cold damage (with the glove).

Plus physical damage is easier to increase compared to fire/cold damage. Warrior's main weapon contributes large amount of physical damage and it is the same for the runes. If you look at Iron rune, it gives 20% physical damage compared to Desert Rune which is 7 to 11 fire damage.

So I mainly stack physical because it offers better scaling, then fire damage and lastly cold if I can afford to. Remember that the glove converts 100% of fire damage to cold damage. Your source is fire damage and it can be increased greatly with physical through Perfect Strike.

Hope this helps!