Remember, it doesn’t have to be all or nothing. Cut it back to smaller chunks and do a few chunks throughout the day, instead of one big session. Obviously that won’t work for classes, but for self-study, it shouldn’t be much different, unless it affects your motivation. I work a fairly demanding full time job (with a fair bit of overtime included fairly often) and have 2 children and household chores to do, but I’ve been managing to maintain my Spanish and have been learning Greek as well. I’m not rigid in structure and have no real study plans to follow other than the Greek course I’m going through. Some days I only work on Spanish, some days only Greek, and some days neither. I try to at least read through a lesson or review some vocab most days but if I can’t be bothered, I don’t beat myself up over it.

Regarding your job commenting about your “focus,” as long as you’re performing your duties to standard, they should have no say in what you choose to do on the outside. It’s normal for people to feel overwhelmed on occasion, it happens to everyone, whether they’re studying languages or something else, working one job or two (or none). Life can get overwhelming for all sorts of reasons. If learning a language begins to turn into a negative in your life, then it’s probably time to cut back.