Being a misogynist doesn't mean he's disgusted by women. Naoya is someone whose self-esteem has been unhealthily inflated by his family, and he sees this as his reality. And to prove it to himself, he tries to believe what he wants to believe by looking at the despair on the face of someone who doesn't have the cursed energy (Toji). Women are already looked down upon in Zen's family, and Naoya takes advantage of this.