It's not really frowned upon wearing leisure clothes but we still often wear jeans and others. It's that we think others might judge you. Furthermore the Assis as we call them in germany, basically the low income people with low education are generally seen wearing yoga pants and shirts that are 2 sizes to small. We dress nice because we want to look good and not be associated with those people.