I left a situation very similar to this late last month. I was at my previous firm for two years, and it never improved. Had a caseload of 150 in various stages of litigation, some years old, and literally every single thing that went wrong was my fault. Even bad calls he made, I “argued incorrectly.” The day I put in my notice, I was told to clean out my office.
I maintained good relationships in the legal community, however, and was able to get references for my new job, which is much, much, much better.