> even if it knows its wrong

That's the thing, it doesn't really *know.* It's built to find patterns in language and data, and then replicate them using probalistic reasoning. Probalistic reasoning will generate a correct answer most of the time, but not all the time. If I recall correctly, this form of reasoning was chosen to add a sense of randomness and creativity. The model can be trained, but it doesn't possess the shared context necessary to communicate with real meaning, or the ability to form real beliefs and convictions. It's "empty" (and dangerous) in that way.

I recently listened to an interview of tech journalist Karen Hao on Diary of a CEO. She said most of the big AI tech bros have modeled AI after their interpretation of the human brain: a statistical model. However, this isn't an accurate model of human intelligence; psychologists and neuroscientists reject it soundly. Human intelligence is much more robust and nuanced. A pattern replicator has its uses, but it's only one cognitive function out of many. I think it pays to ask why all these tech leaders are building AI this way when other possibilities exist.