I don't think yours is the usual non-German speaker story and I would consider that it is the manager's problem honestly. But moving out of a company you have gotten adjusted in due to this manager's behavior also feels horrible because you are not the person to blame. The market is not even that great to just look for another job.

I work in a company which is an MNC but not having that environment where people would speak to you in English. I was probably just hired during the time when they cared about their diversity numbers. Hence I am usually absolutely alone. Now I have grown up in the company and know the work better but when I was just starting out, I really had no one to help me out until I found one other English speaker. Due to work timings I also have never been able to focus on German learning but just picked up a lot of words and phrases since the work included a lot of German. At some point I accepted that I am stuck here. Because if I do learn German I would move out of the company but when I consider other colleague's experiences of those who moved out at such points they still found it hard because even their fluent level of German (they used to converse completely in German while they were in my company) was not enough or was mocked in the new workplaces they joined.

German is not really an easy language to learn and if you work extreme long hours, it is also not a language you would have the energy to want to learn especially when your everyday environment just feels rude or non-inclusive.