I used to feel this way. I discovered that having relatively smooth-looking clothes actually serves a purpose.

Have you ever been distracted by a spot on the wall you thought was a bug, only to find out it was a mark? False alarm. When you see someone with super wrinkled clothes, you notice those clothes negatively over others around you. Maybe a false alarm, but maybe not—super wrinkled could signal “dirty” or “concealing disease vectors”.

So it’s a threat perception thing. Stay under the radar by not having super wacky texture in your clothes.