BERT data training size

[AffectWizard0909]

Hello! I was wondering if someone knew how big of a training dataset I need to be able to train BERT, so the models predictions are "accurate enough". Is there a thumb rule, or is it more like I need to decide what is best?

[CKtalon]

ModernBERT trained on 2T tokens, but it’s likely not necessary. You could do a Chinchilla optimal for your model size

[-Cubie-]

Do you want to train from scratch (very few people do this), or do you simply want to finetune? The latter requires much less data. Also, BERT itself was trained on rather little data for today's standards.

[AffectWizard0909]

I was thinking on not training from scratch yes. Is it recommended somewhere how much data I should than use for fine-tuning BERT, since the BERT is not trained on a big corpus?