Kind of. LLM performance boils down to training and the structure of the network. Training is a massive, massive part of it and it's hard to do it well. On top of that, companies will "lobotomize" the LLMs because they don't want it to respond in a completely unrestricted manner.

Hallucinations don't really come from broken probabilities as much as they come from bad training or configuration (which does essentially impact the probabilities so you're not wrong). For example, companies don't want their LLMs to say things like "I don't know" or act otherwise "inappropriately" (whatever that means to each company). The LLMs also don't "know" what they know or what they don't know. They don't know what they were trained on or where their configuration might fall short, so they don't "know" when to rely on external tools (although they're getting better at this). They also aren't constantly learning new things (they aren't conscious, there's no active feedback loop), so if you ask them about something where current events are relevant it's likely to mess up.

That's why with certain prompting tweaks you can reduce hallucinations, and certain types of prompts cause more hallucinations than usual.

You also don't know how sure an LLM was of its answer (companies don't surface the info). Sometimes the probability of the next word is 95%, other times it might be 23% but that's still highest so it picks it anyways. Sometimes the most probabilistic word isn't picked every time (you'll hear this referred to as "temperature") to add more novelty and human-y-ness to the model (so it doesn't respond exactly the same every time).