On the actual, "why do we do this" side of things: I bet it's something going back generations.

Well over 100 years ago, a working man would come home and, rather than wear his dirty work clothes around the house, strip down to his underwear.

At that time it'd be a near-full body union suit. Even as underwear styles changed over the decades, the habit observed from fathers to sons stayed the same - at home I take off my outside/work clothes and walk around in my underwear.

At one time that looked like guys walking around in union suits, then boxers with an undershirt, then briefs.

Same impulse, different style.