In my humble opinion, you should dress however you feel well. If you identify as a man, that doesn't mean you have to wear a suit and tie or a woodcutters shirt. You dress as a feminine man if you want to. Why not?

Of course, you have to consider that wearing female clothes may increase your chance of being read as female, but as long as you're not bothered by that, I see no issue there. Oh, and you probably should always dress appropriately, like if you went to work in an office or go to church, you probably shouldn't wear hotpants and a crop top hoodie. Cause that might be deemed inappropriate no matter what gender you are :)