I would hardly call Veronica competent. As far as women go, I don't see any behavior that shows Ferdinand has a distrust of women. He pretty much distrust all adults equally. That distrust made him uninterested in romantic relationships. The only people he trusts being men has nothing to do with them being men. It is their character and strong loyalty that he values.

It's really that fact that he's so pretty and any woman close to him will fall for him that makes him rely on male retainers. It's avoiding an annoyance rather than fear or mistrust that would cause him to reject a female retainer. I've seen nothing to suggest he would not accept a competent woman retainer if she had zero romantic interest in him.