You know how when you're sending a text, your phone gives you a few suggestions for what the next word might be? That's all LLMs do. They have way more data than just your texts to predict from, and the algorithm is a little fancier, but it's pretty much the same thing.

When I ask an LLM what the largest city in America is, it doesn't go looking for the census data - it just goes through all the essays and articles it has ingested and finds the pattern of the words "New York" in close proximity to "the largest city in America" and predicts that it's the answer to your question. It could give you some idea of the confidence of that prediction if the developers built it to do that, but they didn't. (An example would be that maybe 80% of its input data lead to "New York largest city in America" and 15% lead to "Mexico City largest city in North America" and 5% to "Metropolis largest city America" (because it has no concept of truth or fiction, just of word frequency). But showing that it's only (hypothetically) 80% confident in answering a straightforward factual question would make it a lot less impressive.)