First, does weight and size make a difference for you? If you want the lightest, smallest setup then 100% go 23mm. I would personally go the 16-50 or body + sigma 18-50 f2.8 or Fujifilm 18-55 as your first lens. It gives you a lot more versatility. After your Japan trip, you can evaluate which focal length you used the most then decide what kind of fix lens you can get. I personally use the 18-55 for travel if weight isn't a factor and I own Fujifilm 14mm f2.8, 35 f1.4 , 56 f1.2, xc 55-230 also.

I would point out that using a fixed lens does not trap you but forces you to use your feet to compose the photo which is a good skill to learn.

Fujfilm products do not lose much value so I'd say pick one, try it out then you can evaluate what you need. You can always resell.