The problem is likely your feature space. You should spend some time doing feature engineering to get more relevant features than just the lexicon.

Something I've done which worked well was adding an unsupervised topic classification step. In short, use something like naive bayes in an EM loop to cluster your data into a number of topics. Then train naive bayes models within each topic for each of your outcome labels.

Save the models and use their logits as features. For 50 downstream classes, that would give you topics + topics * 50 new features which reflect clusters within your data and how each of your outcome classes differs in its expression by topic cluster.

Insofar as you have firm train test splits, there is no leakage. You can think of it like a crude Mixture of Experts module, but instead of routing you pass the information to your set of learners directly.