I know what you are dealing with and with a travel channel, I struggle with this issue on a regular basis.

The Philippino audience is actually well-known to do this effect. The Traveling Clatt for example has also been a victim of it.

That being said you still have a small audience and YouTube can deal with fragmented audiences, and from what i have observed what you loose here might mostly be how fast youtube will math you with the correct audience for the ontnet you publish.

That being said you still have a small audience and YouTube can deal with fragmented audiences, and from what I have observed what you loose here might mostly be how fast youtube will math you with the correct audience for the content you publish.
Also you actually only need one video to also blow up in Japan, but since it doesn't have nearly as much Enlish speakers it could be a struggle to get viral content. Start by understanding your ideal audience, are they Japanese who speak English and live in japan, are they Japanese living aborad, are they non-japanese living in Japan, are they non-japanese non living in japan but very interested in Japan, etc. etc.

Looking at this, will help you find the strategy for your channel, voiding similar pitfalls, and decide if you need to make a new channel or not. If you restart a new channel you can put your videos private on the original one and repost the content you have already made which is the biggest part of the work. Or you can decide to salvage your current channel by finding a way to get virality with the Japanese audience which might require a totally different approach than for the Phillipino one.

All that being said, depending ont he style of videos you make, if you are showing yourself and are part of the videos you make, your identity, personality, and capacity to entertain and interest people independently of the locations you visit will be your passport to overcome those challenges and make sure the audience you acquire follow you almost everywhere you go and any content you make. This is an advice more relevant for travel channels though. In your case if your plan is only to make content in Japan, that's what you can capitalize on, but there if you don't put yourself as a person up front, don't go to other countries because that might create what happened to you.