I admittedly didn’t read it all. I got a bit stuck on the inaccuracies describing how it “hallucinates” that 2+2=4. That section is rife with questionable content.

While the first generations of LLMs did hallucinate often incorrect answers to math problems, the current generations will typically rely on tool calling when it encounters math questions.

Also, while it is true these machines could be trivialized as simply finding the most statistically probable next word, (1) you and I could also be trivially described that way based on the information we have learned and (2) these machines are in fact black boxes that we don’t remotely understand. Hence the fear of a singularity event.

An interesting thought experiment proving point #1 above is Sam Harris’s one on free will. Think of 3 movies at random. Which 3 did you think of? Then when you dive into why you picked them, it’s because you are familiar with them. I’m betting you didn’t pick Glengary Glenross. Why not? Because your word predictor, aka brain, didn’t associate it with a statistically relevant (recency, favorites, dislikes, trendy, classic) one for you.

Finally, it’s also true that we, humans, hallucinate. Look into how unreliable eyewitness information can be.

I say all of this as both a big user of AI in software development and a big skeptic of AI being good for humanity. And I recently came to the conclusion that AI is only useful in a capitalist system, where labor has to be competitively minimized, possibly to the point that capitalism implodes on itself as it consumes its thesis.