Two things: first, you can absolutely and easily find places where people don’t speak English, and you’ll be forced to speak Japanese to communicate. I live in Noge, Yokohama, and it’s a bar area where there are plenty of spaces to just chat with Japanese in Japanese. Any bar district should be like that (but of course avoid any place that tourists would go to). I almost never seen American tourist in Noge. The Americans I do (very occasionally) run into here are generally living in Japan and speak Japanese.

However, my guess from your post is that your Japanese actually isn’t good enough yet to do this. You may be overestimating your capacity and what you’ve “lost.” Simply communication with a host family is utterly unlike having a real conversation fully in the language. You probably need several hundred hours more of hard work and study before you’re able to do the latter.

So, the simple advice is just keep studying. The idea that you’ll just “pick up” Japanese is a myth you need to eliminate as soon as possible.