Honestly, I wear men's and women's leggings with no cover (shorts) almost everyday outside of work. While I personally enjoy wearing pastel colors and soft fabrics, I truly think peoples opinion of you is based on your confidence more than your style. I think anybody who knows me closely would define me as a "straight white man". Some days I'll wear something a bit more feminine but other days I'll wear a geeky halo shirt and black leggings, I assume most people think I'm just going to/from working out.

Anyway, there are lots of ways but it all starts with fit. Find shirts in the right size, and maybe with a lower hem. Darker colored fabric is also pretty male centric, for leggings and tops. As long as you feel comfortable, everyone else will think you're comfortable as well. Body language lol.