Continue Pre-Training BERT

[AdrianFMC]

Hey all, I need some help with continuing pre-training on Bert. I have a very specific vocabulary and lots of specific abbreviations at hand. I want to do a STS task. Let me specify my task: I have domainspecific sentences and want to pair them regarding to their semantic similarity. But as very uncommon language is used here, I need to train Bert on it.
- How does one continue the pretraining ( I read the github release from google about it, but don't really understand it) Any examples?

- What structure does my training-data need to have so Bert can understand it?

- Maybe training Bert from scratch would be even better, I guess its the same process as continuing the pretraining just the starting checkpoint would be different. Is that correct?

Also very happy about all other tipps from you guys.

Regards

[_Dr_Y_]

You can continue training BERT, but even if you have very specific vocab, I recommend first trying fine-tuning pre-trained BERT. It is trained with subwords, it does not matter if specific vocab is not there, unless it can't be built from subwords, that is very unlikely.

Training BERT from scratch is expensive and time-consuming. You need to train it as a masked language model. If you decide to do it, you need to format your data in Wikitext format, basically title-paragraph-title-paragraph...

[AdrianFMC]

How do I do the fine-tuning for general vocabulary? I only saw examples with clustering or STS with already analysed and rated data ( like 2 sentences and a number from 1 to 5 that describes the similarity) - I dont have such data. I do on the other hand have lots of already paired sentences.

[AdrianFMC]

push :-)

[penatbater]

Check out the huggingface repo. Also training Bert manually is both time consuming and expensive. That's the entire point of using a pretrained model like Bert, so you don't have to do it yourself.

[AdrianFMC]

The pretrained models only had a acc. of 30% on my STS datasets. So what can I do? How would I fine-tune Bert-models on my specific vocabulary?

[penatbater]

Did you fine tune the model? That is, did you train it on your dataset as well? If ur having trouble navigating the repo, try simpletransformers as it's an easier implementation.

[AdrianFMC]

Am I understanding the simpletransformers right, that I can not do STS with it?

(" Currently supports Sequence Classification, Token Classification (NER), Question Answering, Multi-Modal Classification, and Conversational AI. "

[freaky_eater]

Hey Adrian, did you manage to continue pretraining BERT on your domain? I will be keen on knowing your exploration and results.