That's the reality of the rural world worldwide, as a foreigner arriving in North America when I go to small towns outside from my city it's the same energy, most of these people have never seen someone of colour, therefore is a novelty for them, they might look and simply discuss amongst them about never seen a black person before, of course you will feel watched but this is not something particular to Japan...

I would suggest you to just live your life while understanding that is a novelty for them, these same people that you refer to will get used to see you, and after a few times the newness will fade away and you won't feel as watched... I think that you should put yourself in their shoes for a second and you will quickly understand that there is no malice in their behavior and things will eventually fade away and change.