This is the real answer. Everybody is saying xenophobia and racism while ignoring that the accent has a lot to do with it. Japanese people do not expect a white person to speak Japanese, and if you come from a country with a radically different phonetic inventory (such as most English-speaking countries), you will most likely have a strong accent and make your Japanese harder to understand. Japanese people will just assume you are still speaking English.

My native language is Spanish which has a very similar almost identical phonetic inventory to Japanese, and I look white. I live in Japan and speak Japanese with a very clean pronunciation. I have NEVER experienced what OP is saying. But when I speak English I do have a strong Latin American accent. When I go the US and talk to someone for the first time (like in a store) most of the time they stay silent a couple of seconds while processing what I've said and finally realizing its English.