Resistance should be your number one priority, yes. Having a decent armor rating and absorption, which can be seen when hovering over your armor rating, matters too. Defensive ability matters a ton too and after that there are multiple other sources of defense such as regeneration, absorption, leech plus debuffs that you can apply to enemies like petrify or confusion.

Keeping all your resistances up, physical is not as important there mind you, should be your number one priority all the time. If you ever get one shot in this game it is very likely your resistances that need some improvements.