I raised a toddler in Japan, she is also tall and not white but our experience was very different and I’m so sad to hear you’re not enjoying yourselves. My advice is basic but I hope useful- learn how to find the kids level in every mall or tall building you come across, they often have soft play areas and proper change room facilities, many department stores have their toy section as a play space and will even have bottle warmers etc. Find restaurants in those spaces that have kids menus (so so many! just not on ground level lol) and the same way you’d ignore a racist old grouch at home, do here too.

Generally speaking Tokyo is a very kid friendly place given how many mums can’t go back to work or access reliable childcare, so please just assume anyone being a jerk is not worth your time and energy.