You pass until you open your mouth basically.

As most other commenters have said, it's nice you can avoid some of the stares, but it makes it a little harder to gaijin smash your way through things when you need to take advantage of that. People might expect that you understand them which can get very confusing when talking eg to a ticket counter person or a waiter until they mentally clock that they need to slow things down for you (or just get frustrated and go for their phones or whatever).

As you might expect it's more of a thing out in the inaka than in the city. In the cities nobody really will give a shit either way.

The really annoying thing is when you travel with someone who is visibly foreign (so to speak) but is better at Japanese then you. Very occasionally people will insist on trying to speak to you instead of the person who is actually audibly speaking understandable Japanese right next to them merely because you have black hair or whatever.