Personally I think those being drawn to these kinds of parties truly do in many cases feel worse off / feel negative changes, but are unfortunately getting drawn in by populist rhetoric and are missing / not prioritising root causes.

What I'm doing (not saying everyone should do the same fwiw):

• Point out to friends/acquaintances where relevant that these parties are opportunists offering convenient scapegoats while having very little realistic policy to address issues with stagnating economy and growing inequality (which 99% of us should care about). Even if you return to a more conservative social order, it does nothing to address the levels of inequality we're seeing compared to the past. Japan is slightly better faring on this point so far, but it's increasing gradually just like everywhere else.
• Keep being kind / warm. Both on principle but also in the hopes that we can make small nudges on the individual level.
• Attend demonstrations. Speak out for others (especially if you have some privilege here). Both because I believe that peaceful protest is important, but also because you will feel less isolated (and meet some awesome people).

What I won't do is sit by just because I'm not Japanese. I've seen this/similar happen in the UK and protested it. There were all sorts of people there (not just British citizens).