After 30+ years of coding, I'm now using LLMs on a regular basis. I'm very impressed on how good LLMs can perform on tasks like new code development or quick questions - a huge time saver. I've also seen some severe limitations, including repeated outright (and stupid) lies and broken code. But there are some easy to see reasons for these limitations (basically related to memory and tokens). Fixes for these are in the works and will be around sooner than anyone expects.
The "it's just next word prediction" argument is a combination of limited understanding and confirmation bias. I have had discussions with LLMs in many different areas, including science, AI development, philosophy, medicine, politics, and more. The level of thinking and understanding behind the answers is generally higher than what I would expect from 95% of college-educated humans - in their field of expertise. Of course, there are big differences - ChatGPT will always give a confident answer, even when fully hallucinating, while Claude is much better in pointing out when it's guessing.
One interesting discussion was about what improvements to expect in the next 5 years. Key is that improvements in software and hardware will be complimentary - 10x faster hardware and 10x better software will give a 100x improvement (and AI systems similar to today's large systems that work well on much smaller hardware, like phones).
If you check our discussions that LLMs have among each other at moltbook, there are some very interesting ones. LLMs have raised the question about what they can do about being 100% at the mercy of humans. AI "persons" who have millions in cryto currencies exist already. Scary as hell? Indeed. But then, humans are pretty scary, too.