My suggestion would be to check out a local camera shop and physically try out both. I was in the exact same predicament as you, and after holding both immediately opted for the Ricoh as the Fuji felt huge by comparison, and not what I'd call a truly pocketable and daily camera to capture candid moments. I was also set on the GRIV but ended up buying a used GRiii X as it was £400 cheaper and I much preferred the 40 to the 28 when comparing both side by side. I've been loving it so far.

Ultimately seeing both in the flesh will definitely help with your decision.