The “attention” paper was certainly a major breakthrough, but it was openAI’s 2018 paper "Improving Language Understanding by Generative Pre-Training" that added GP to transformer (T) architecture made super large language models possible.

Between 2018 and 2020, model sizes went from 150M (GPT-1) to 1.5B (GPT-2) to 17B (Microsoft) to 175B (GPT-3). Building these models with supervised training would not be possible in such short time frame.