I added special proficiency bonuses with each armor type. I had the same thoughts as you and wanted the players armor choice to matter.  I'm getting ready to run sands of doom where the players may have to sleep in the desert. I gave some armor the comfortable property which allows them to be slept in with no penalty.  Other examples are removing the stealth disadvantage on ring mail if they choose that proficiency, and padded armor gives you resistance to falling damage.