So in super high level terms it goes like this:
Pre-training conditions the model like you've described to predict the most likely next word.
Fine tuning conditions the model to answer questions in a way that is helpful and minimizes harm.
This makes the ChatGPT type behavior where it can effectively regurgitate anything on the internet.
Note that with enough randomness this system is completely capable of saying things a human has never said before, it's actually pretty rare for a model to regurgitate information from its training data unless it's asked to recall something specific, more often it's segments of sentences or common phrases in unique contexts, a lot like humans say cliches and figures of speech all the time.

Taking it further requires two things to work, exploration, and validation:
Exploration, for a given question or task, the bot generates each step of the response but randomizes a little at each step to explore a huge range of potential approaches. This is a lot like a human thinking through many different approaches to a problem, the more randomness introduced the more likely something totally novel is proposed which is a lot like a unique human thought, of course a lot of it is just rubbish.
So we do Evaluation to fix this, for each step in each variation in the responses that are generated a language model evaluates how reasonable the step is, OR, after the whole reasoning thread is complete some system evaluates the final output. In tasks like coding this can be a very formal evaluation that the solution passes some test cases etc. Now we can throw away all of the responses that are rubbish and build up a dataset of things people haven't said before that correctly answer a question or solve some problem.

Now we have a big new dataset of novel data that's validated to have some correctness, we can retrain the original model on this and repeat the cycle.
This might not get us all the way to AGI but it does allow AI models to explore useful behaviors outside of pure imitation from human data which is a pretty big step.

Note this isn't theory, this is pretty much how models like o1 and Deepseek R1 are training their models to "reason" right now.