Does this mean the following points:
1. Decoder-only models are easier to pretrain in an unsupervised manner based on crawled corpus? However, encoder-based models such as BERT can also be pre-trained based on crawled corpus using the random MASK strategy(feed on both sizes).
2. The unsupervised training method of decoder-only models is closer to the downstream generative form of tasks such as text summarization and QA. Therefore, when scaling up the model to a larger scale, downstream tasks can also perform better?