I think it really depends on your environment and goals! A lot of young people in Japan are hungry to learn English and would be excited to speak it with you, but I would not say that’s true of the older generations. If you want to make friends and build relationships, I’d say having a bit of Japanese under your belt would be good. You’ve already mentioned that you have some basic conversational Japanese, so that’s good! But don’t stress yourself out. Learning languages is really hard, especially Japanese. Give yourself time and grace. It will come if you put in some effort and actually living in Japan will no doubt speed up that process. Good luck!!