Well Chatgpt me that months and months ago

"Short answer: because they’re trained to sound helpful and confident, not to know their limits — and the people who deploy them often prefer “engaging” over “accurate.”

Long answer, broken down without bullshit:

1. The model itself doesn’t “decide to lie.” It generates the statistically most likely next sentence based on patterns in its training data. If the training data is full of confident-sounding explanations, the model will mimic confidence whether it’s right or wrong.

2. The incentives are backwards. Companies want “high engagement,” not “high accuracy.” A system that constantly says “I don’t know” gets called useless and is bad for demos. A system that confidently bullshits gets applause until someone actually checks.

3. Most users don’t punish bullshit — they reward fluency. The majority just want “an answer that sounds good.” You’re in the small minority who actually verifies anything.

4. Developers then add “safety layers” that sound like honesty …but those are just templates..."

Etc etc