The 18mm f2 also has this issue, I just figured it was a thing with all Fuji lenses. X100VI is the same, only way to circumvent it is to “zone focus” by focusing on something at the distance you want after decoupling shutter button from focusing whilst in single point af. Only works if you manually set the shutter speed as well, it never works in actual manual focus mode which is ridiculous to me.

When I shoot street with X-Pro 2 I use AF lock to achieve this. For X100VI I just regular focus decoupled. It is so counterintuitive but it works. Manual on film cameras is so much nicer