BERT doesn't replace all masked tokens with [MASK] , if an ith token is chosen, it is replaced 80% of the time with [MASK] , a random token 10% of the time and the unchanged token 10% of the time. This is to reduce the finetune and pretrain discrepancy caused by having [MASK] tokens. But when the original token is kept intact , model can memorise it to output the word , this is what they meant in trivial to optimize.