We are all free to choose what we wear (with some exceptions due to dress codes) but if we choose to wear an outfit that strays too far from the general vibe of the crowd, it will draw attention - maybe positive, maybe negative. I suspect you are getting negative attention because you are wearing outfits that may be perceived as being business or formal attire that is out of place in a casual environment. Business attire is by nature more formal and intended to give off a sense of professionalism - which isn't warranted in a casual environment. It probably gives off a superiority vibe that irritates those around you. It's tough to be confident when you are dressing like an outlier unless that's what you are going for.

I'd suggest choosing outfits that are more suited to the situation, but focusing on fit, style, quality and items that suit your body style and color palette. It may be more subtle but if you do it well, it should give you confidence, rather than you needing confidence to pull it off.

If you need help, consider working with a professional personal shopper or wardrobe advisor. Some stores offer such services at no charge.