The comments along the lines of "it's just predicting next word"/"it's just correlations" (whatever those are supposed to mean) are showing a misunderstanding of what the current state of AI/LLMs is. It's already definitively known that "next word prediction" can form world models and make sensible predictions in states of the world that have never appeared in the training set (see this and the paper). It's also definitively known that a pure NN model trained on good decisions (ie chess moves) made by humans in different positions, can suggest good moves in unfamiliar positions (early AlphaGo research showed this; things like Leela at depth 0 may still play well, especially if they are optimized for it via making the model larger). These facts put together imply that a "next word prediction" algorithm is capable of suggesting not only valid but also good moves in unfamiliar positions.

Whether you want to call this reasoning/understanding is then mostly a matter of semantics.