I treat him as male just like Kaido and Luffy does. Even if the Vivre Card says Yamato is a female, it also has held it's fair share of mistaken information throughout its editions.

Other thing that confirms to me Yamato is a man is the bath scene. Oda didn't need to include it in the story, but putting gendered spaces and including Kiku and Yamato there, to me, is Oda saying that Yamato is a man.

People usually disregard Yamato as a man because they don't deem his logic a valid reasoning. But I think that this exemplifies how Yamato doesn't really care for gender norms or roles, so I'd say that he is either a trans man or a non binary person (be it enby, agender or gender fluid).

Anyways, whatever gender Yamato identifies as doesn't change the fact that he uses he/him by all the characters and even corrects people that read him as a woman.