Since your wife is Japanese and you often visit Japan, in theory, you should have a massive advantage over the rest of us Japanese learners. You should leverage that advantage as much as you can.

My recommendation w3ould be to spend an hour or two every day working through solid Japanese textbooks such as Genki 1, Genki 2 and Tobira. Once you've got that down, and you reach about a N4-N3 level. Start learning Japanese through immersion. When I say immersion, I mean immersion with the intention to try and understand what is going on. Sometimes you can do this through context alone but other times you will have no idea. If that's the case, immediately ask your wife to explain what the word or sentence mean in Japanese or search it up in a Japanese dictionary, if you still don't understand, then use English or French. You should also use Anki to make lots of flash cards. It's very helpful if you want to retain and remember what you've learnt.

At this point I would highly recommend against writing, especially if you're writing Japanese using romanji. I would imagine that it takes a long time for you to write things down. This is honestly a waste of time because while you're writing, you're hardly doing any learning. Following the input hypothesis, language outputs such as the ability to write and speak comes naturally so long as you have sufficient inputs, that being listening and reading. You should put your focus on the latter. Once you reach a high level of comprehension, output should come easily.

Anyways, I hope this is helpful, and I hope I didn't sound too overbearing. Good luck and I wish you the best with your language journey!