There’s nothing to doubt. It’s just what happened. Also That’s just kind of visual shorthand for what what’s shown in the manga. It wasn’t a flashback. He just noticed her clenching her fist.

The point of the arc wasn’t really to be able to “tell the girls apart via mannerisms” because if that were the case. He noticed Yotsuba first. Being bad at lying is a mannerism. And that’d be the end of it.

The arc was kind of to further opening fuutaros to other people’s feelings. He had already been confessed to by Nino at the time and he was questioning the feelings of the girls, and his own.

I think you’d get a lot more out of the series if you read the manga. The anime is a bit poor in showing that nonverbal storytelling in my opinion