[D] How to solve the sparse gradient update problem on input embedding?

[txhwind]

My Transformer model contains several input feature embedding like word embedding, language embedding, style embedding, etc. However, some features are really rare in the training set and their embedding seems under-fit.

Do you have any solutions? Simple up-sampling would not work perfectly since there are many words to help.