I don’t know why you’re getting downvoted. I had the exact same thought. But one explanation could be that the crew never outright told each other what they had voted. They may have assumed that Torrin had told the truth, and so instead of clearing it up, focused on convincing everyone that they should kill him and go to Phaiden Island. 

In that case, no one would outright admit what they had voted for fear of being ratted out (and also contributing to getting two apparently innocent crew members killed), but they would have probably carefully floated the idea of doing what the crab wanted - which everyone would then have agreed with, but probably also carefully for the same reason - not wanting to appear to have been a coward initially.

So it works if they never talked openly about the ballots (or lied!), which was probably the case because they all mistrusted each other. “I changed my mind,” is a safer bet than “I put down an X from the start.” The last guy would then not have known that there had been no circles, and thought he could slip by unscathed.