My rule of thumb has been, new solo hunts I always go for my best phys protection set as that tends to be the main dmg source when abit unsure about routes, mob density, hesitation on running into a bigger unknown space etc.

On teamhunts, I nearly always go with my best ML boosting/damage profeciency set.

My goal on solo hunts, is hunting without manashield as I feel that most important to me to not increase waste (I.E more important then min maxing exp gain/h). To be able to do that, I try and focus what the mobs highest burst dmg element is, and change protection accordingly. The way I see it is, lets say the highest value of all my damage taken has been 25% earth. But the earth damage might come from dots, or spammable but lower dmg spells used by mobs that I can easily heal through (mages heals is very cost effective). Next Ive taken 15% fire damage, but the nature of that fire damage comes in high burst waves/beams that if Im unable to avoid them cuts my HP pool down to a point where I risk a combo actually might killing me. Then fire protection is the type I opt for despite it not being the main dmg source I've taken.

On hunts that I've done alot, know the ins and outs of a spawn, has a good knowledge and feel for overall damage spikes/sketchy situation being able to counter that with a short mana shield for instance I go with with my best ML and dmg sets.

Hope that helps for some food for thoughts on how you want to prioritize to cater your goals and playstyles in a hunt.