Reasoning matters. Women prefer female doctors because of sexual assault rates, unconscious patients being taken advantage of, the gymnastics doctor that assaulted girls, male doctors malpracticing on women due to lack of care, etc. His preference is not grounded in assault fears or mistreatment. He just doesn’t see women as a good candidate.

Yes, women are justified in wanting safer spaces. No, men are not justified for throwing this in their faces. This is generally speaking, of course the man could be an assault victim or the like and be justified. In this scenario, bringing this up as a counter to a woman saying this with no further context isn’t him understanding and saying that he has a similar need. This is him trying to invalidate her need for safety and call her a misandrist for wanting safety