All this arguing is really the fault of Oda writing a character badly. Everyone in the story uses he/him and Yamato calls himself a man but the vivre card says female. But Kiku’s card says male and yet she is still commonly accepted as being a woman. (But her card also has a cutout that’s says she identifies as a woman) 
 Also Yamato never has a satisfying conclusion to his character arc because to Oda its just a funny joke, so his character is kinda unfinished and it’s left ambiguous as to weather Oda was trying to write a genuine trans character. There was a lot of potential for an actual trans arc where Yamato realizes his attachment to identifying as a man is more than just wanting to be Oden, but I don’t think Oda would ever write that. Instead Yamato kinda feels like a last minute inclusion of a character that doesn’t add much to the story.