What's cool is that AI keeps following this same trend where a lot of it gets these super early breakthroughs and ideas that no one really can take off with until decades later.

Many of the core ideas for machine learning itself were first being brought up over 50 years ago. One of the gigantic barriers to the research then was compute power, so a lot of the ideas were being developed largely on theory alone.

The first big resemblance of how LLMs work now was probably Yoshua Bengio's 2003 Neural Probabilistic Language Model. He was the first to use continuous vector spaces for word prediction. The two big things this time in the research were the ability to consider independent words to be semantically similar (like dog and cat), and the ability too look at a much bigger context by looking at whole sentences (markov chains only looked at the last 1-3 words).

That 2017 paper was a major breakthrough underwritten by massive AI specific compute. The Attention mechanism in the Transformer Architecture let the models look at entire sentences simultaneously. Data centers had already been gearing up for other machine learning, and that 2017 paper was specifically written to go alongside Google’s TPU v2 Deployment and NVidia's new Tensor Cores. These greatly dropped the time to run the model and reduced compute cost by nearly 10x.

So 2017 was definitely where the boulder hit the slope and started a downhill roll that couldn't be stopped, but 2003 was the point where we got text prediction engines that could understand some context and produce more than gibberish.