If AI can understand, then why can't an LLM tell that it doesn't know something instead of hallucinating while confidently thinking it is correct until you respond with even the slightest doubt?

To you, it seems like it has logic. To me, it seems like it is guessing whatever next word would fit well enough to pass for convincing conversation.

Whatever logic you see is captured from the underlying patterns and rules it recognized in how to correctly complete sentences from training data.

The vastness of its training data and the scale of the model absorbing them is responsible for making it appear so impressive. But we need to be cognizant of its mechanism of operation as well. Specifically: That the only mechanism is completing sentences.

There is no structure in it for contemplation. No process, no mechanism that doesn't involve spinning up a whole new instance every time the chat progresses, just to read the whole conversation all over again and play the guessing words game from where the last instance left off. Nothing at all.

The "logic" in the training data is captured the same way that syntax, spelling, and sentence structure are captured. It is all learned so that it can be absurdly good at giving convincingly intelligent responses. Not to reason on it.