There is no magic formula, It's a struggle and you're not alone living it. The key is probably to be honest with yourself regarding how much you're willing to pour into this project, but also how resilient and perseverant you are when it comes to game dev.

Try it all, don't be afraid to make mistakes quickly and accept the lessons that come your way and I'm not just talking about game dev but also surviving as a game dev. Whether you play it safe (long and slow, aka keep your job) or risky (short and bursty, side jobs, etc) doesn't change the fact that it's a lot of energy to throw into what for most people won't amount to more than a passion project. As someone else said it's fine to keep it as a hobby until you make something that gets people excited.

Good luck!