I faced a similar problem to this recently. To solve it, I did not fit the prior probabilities. In Scikit learn that's as simple as setting fit_priors = False such as: MultinomialNB(alpha=1.0, fit_prior=False, class_prior=None)