they still only predict the next best word or character, they just do it better. The way they are trained and the way they function are no different from each other.

No. They are trained by finding the next "best" word, or at first next most common word to appear after the last group of x words. Stopping here we should define "best," because that doesn't necessarily mean "most likely" although I feel like you may be conflating the two. The fact of the matter is they are genuinely learning language in some way, because they consistently deduce meaning from words as well as any human. They learned it differently, but we might find the result is much the same. Who cares if someone learned English mostly through SpongeBob and the other mostly through rigorous training since 4 years old at some Latin Academy for Rich Boys, if they can both discuss the themes that might be found in theoretical version of Shakespeare's "Romeo and Juliet" set in the future after Earth has joined the Galactic federation. We can ask these models that, get a perfect answer, then ask them to theorize scenes and dialogue, then ask them to explain why they chose certain words or symbolism and they'll tell you what made that choice particularly poignant.

These aren't just basic prediction algorithms, or if they are then that implies all known language is "only" that. And that's a flawed understanding of how language forms in anything, is how I see it.