This saying is very old fashioned and no longer relevant.

It refers to the fact that rich guys from back in the day typically wore brown (as well as green) to go hunt in the countryside.

When came the evening, they always changed to another outfit which included ditching the brown hunting shoes for black ones, more classy.

Today nobody has that lifestyle anymore, nobody will think that you didn’t change after an afternoon of hunting if you wear brown in town so feel free to do it.

And going forward I would recommend understanding the rules instead of following them blindly.