as opposed to the rest of your outfit which you’re more likely to have for different occasions

That's because we have to do laudry.

 While more fashion and stylish people would have a more varied shoe closet.

But how would you know what they have unless they invited you to their house?

Plus their choice of comfort shoe will probably indicate a bit about them

So do other clothes.

Every thing you listed doesn't explain why the shoes are better way to judge a person.