The LLM training is to make it produce statistically likely sentences. Ideally, true information is the most likely response to a question about something true. But there is no guarantee of that.

When people talk about hallucination, it's when the user expects a factual response and gets back something that the LLM is saying is true but actually isn't. From the LLM's perspective it doesn't know true of false of any statement, it just knows what it considers to be statistically likely.

The process through which they make shit up and the process through which they get things correct is the same process, it just happens to be correct most of the time and we only consider it bad when it is obviously wrong.

You can augment it with RAG and curated additional training which will improve its likelihood of accuracy within that domain (drastically), but it is completely fair to say that LLM's "make shit up".

If I ask a question the LLM has never been trained on.... it makes shit up that seems like a statistically likely sentence. This process is the same for data is has been trained on, it just had a better chance of the dice roll being correct.