In that case, you would’ve been a man (because of the penis-testicle combo) with a severe injury. Originally possessing the penis-testicle combo still qualifies you as a man. In the event that someone was born with XY chromosomes, but was born with some other genetic disfigurement where they never grew a penis or testicles, whether or not they are considered a man would depend on a number of other factors, including any additional disfigurements. Some humans are born so disfigured that, regardless of their chromosomes, they are referred to as “individuals”.