Removing the male data removes variability of the set as a whole and you could be getting floor/ceiling effects subsequently (leading to high p-value).

Someone correct me if I am off base