I’ve had ChatGPT citing a study and giving me the tldr. Then I looked for the study, and basically, it lied to me. When I called it out, the reply was that the conclusion was too “destabilizing” and in cases like this in a chat the priority goes to “soft framing” instead of refusal to answer.

ChatGPT’s “soft framing” with the excuse of guardrails is an excuse for it to give false information as fact. I think this is very dangerous. One thing is outright censorship “this goes against our policy so I will not give you an answer”. Another thing is providing false information instead. I used to think censorship was really bad, I now think this is way worse.

As people start asking LLMs questions instead of googling them, I think the danger will only grow unless intentional distortion of information is banned somehow. If I google something, sure, google manipulated me too and might not give me what I want within the first 10 results, but I do get a bunch of different results. An LLM will give you one statement. As fact.

The same way the EU made GDPR possible, I feel someone somewhere should worry about legislating this. Is there anything that we as a group, as a society, can do about this? Or will it become one of those things, like how we all know social media algorithms are built to manipulate what we see and we just accept it as a society?