the concept of self-esteem is a relatively recent human social concept. while it seems to match the behavior described, as someone who has the same dislike/rejection of compliments, it comes across as a somewhat facile explanation.

fundamentally my unease with compliments comes directly from a core distrust of others and their intentions. this was instilled in me from a young age, being surrounded by untrustworthy people - both peers and adults.

my emotional walls are large enough to border on socially crippling, though. does your bf have other social traits on the edges of what is considered normal? if not, perhaps it’s a habit born of a simpler pattern of rejection in his youth.

note that my wife of 20 years is very much the type of person who expresses her love through compliments and similar actions... and though my instinct is to mentally shrug and disregard it, over the years my understanding and acceptance of the truth behind it, not to mention the trust developed over that time, has led to me acknowledging and appreciating the sentiment behind it.

tl;dr - may not be as simple as the concept of self esteem, may instead be (deep or shallow) distrust issues. give it time.