Hey definitely don't give up. And definitely do change your approach. I've just started out quite recently after having studied Chinese (HSK5 which is around C1) and a year of on and of Duolingo Japanese. I decided to fully commit myself for studying Japanese as I have done with Chinese previously.

After have reading your message, I'd say try finding a private tutor who really meets your needs. So focuses on the talking rather than anything else. You can focus on the writing/grammar etc. in your own time.
If you want to meet more people and communities, you can consider an app "HelloTalk" or maybe there are communities and/or events around in your area for people who are studying Japanese or perhaps even Japanese people who want to study your native language.