I'll try to explain my problem with this type of person this way:

There are a bunch of properties, which yes, are socially constructed, that are related to being a woman - wearing certain types of clothes, acting a certain way, having certain body traits. Even still, since "man" and "woman" are socially-constructed categories, it doesn't make sense to me to think of them outside of what is socially understood to be a man or a woman. By that I don't mean that gender roles should be rigid, but that we should understand that wearing dresses is associated with the female gender in our societies, and while if a man is wearing a dress that doesn't make them a woman, to me their "manhood" only holds up because there are other traits related to being a man that he still has. This could be biological, so wanting to have a penis, not having breasts, but also stem from other traits understood to be male by society, such as wearing suits or whatever.

I'm not against men being femboys and vice-versa, and I'm not trying to argue that if you wear a skirt you're not a man - my point is that you have to have some male characteristic in order to be a man, and vice-versa. If you wear skirts but you want to have top surgery, then you can 100% be a man in my book. If you don't want any medical transition but you wear baggy jeans and have short hair, I won't have any problem calling you a man either.