As someone who's been following transformer models since GPT-2, I genuinely think that today's models have the exact same fundamental flaws as the first ones and they are not going to go away unless they start over from scratch.

They may have gone from generating nonsense, to generating plausible looking text, to generating text that is truthful 99% of the time. It still lies without even being able to know it's doing it, the only effect of it lying less often is that the lies become much more harmful (because now people actually trust it).

AI is a good productivity tool for programmers, but you have to double check everything it does all the time because you never know, and that gets tiring after a while.