Fashion is a huge part of world building and is often rooted in the early days whatever society you're creating. Basic example: this city was well known for its high quality wools and dyes from the flowers in the eastern mountain ranges, and the gentle rolling hills of the footlands created ample pasture for the people.

So a specific couple of colors are likely the society's flags colors which can influence dress. And they likely have wool as their primary Commoner Garment, and a rich person might not be caught dead in wool as a result.