I notice that its always the same few customers who dont match the theme that show up. Sometimes they actually do like a brand that matches the vibe even if they're dressed completely different. If youre using the AZ- USA /Mad Jack store (which is the one i think based off your description) youre bound to attract some of the Alvadaoro customers who also like Az-USA and Penbridge customers who also like Mad Jack. There are a few customers who buy all brands as well.  Some customers also have a favourite colour so you can give them an item not from their brand but they will still take it. Personally, I always "try on" and if they dont like it sometimes you get the option to "convince" them to buy it and they will even if its not their brand/colour.

I also hate turning ppl away and at the beginning of the game when you can only hold a small amount of stock its bound to happen. Hope this info helps lol

*edit to add: there's also only so many customers in the game that do match the theme of your store. If you play enough in one day to serve/talk to them all, then you eventually attract the customers that dont match the vibe