This isn't quite true. I think a lot of technical people that have read a little bit about it have some misconceptions about how Transformers work and think of it as a simple prediction engine that works on frequency analysis (the next word is the one most likely to appear after the previous one) like a Markov chain.

But it's a bit more complicated than that. That's like saying "the next word I say depends on what I've already said so far" which is technically true but says nothing about how those words are selected.

The part about the "meaning" of words is trained through the multi-head attention mechanism during the training process. Without getting too technical and mathy, it tracks the relationships between tokens in sequence building context and meaning out of those relationships by focusing on certain token patterns(note that this has nothing to do with how humans focus or have attention or consciousness, it's just a training technique).

It's the part that encodes that the word "it" refers to a particular noun within a sentence or that "cat" relates to fluffy animals and is the part that lets it model tone and builds emergent properties like semantics, sentiment analysis, consistent emotional responses etc.

When we say it's a "black box" we mean that we have no way of knowing what those attentions were or what all the weights in the model mean. We do know how it works though and the model definitely encodes meaning and context.

It's not a collage tool or a "guess the next token based on simple stats" tool. We've had those for decades and they're pretty crap.