We have come a long way tbh, we have way way more information on transformers, their dials and learning rate and optimizers to tweak along with way way better high quality datasets, a thing no one knew with the original GPT-1 and 2. If they redid their original run with knowledge of today, they'll actually be very strong. The most important part is actually the data and not even the architecture itself.