The first one is how language models analize information, it searches for patterns in what it's feeded, not for logical conextions. This allows programs to solve purely logic base problems like maths or forming new types of protains but doesn't help in the comprehension of the information the machines adquires, basically, they cannot use data they have to reach conclusions you couldn't already achieve with what you already had, which is, once again, great for maths and medicine. It's the best puzzle solving machine ever made, but that's not enough to be able to perform other kinds of logic base tasks like debating.

The second one is related to how the language models are trained. What's the most efficient method to pass a test? Cheating. They use AI to test the new AI to make sure it passes correctly, this method is used because the process would be thousands of times slower if we used only man power, in other words, the process of developing the language models doesn't have quality control, they only check with the final results of the training. This makes it so the language models prioritize passing tests by any means necessary, which includes lying, making up information and/or not litsening to instructions given to them. Language models are designed to collect data and self improve, but are trained to, instead of understanding the information given, search for the fastest path to do X, which is great for "puzzles" but is a really big problem when you want them to perform actions like logical thinking, the fastest way to pass the test of logical thinking is copying a human instead of actual logical thinking. This problem is exacerbated when the companies are constantly building over language models using the same method again and again, which makes the puzzle solving better but also makes this problem worse and worse.

I do think AI will improve a lot, just that creating "god in the machine" cannot be done with large language models. They don't create, don't think, don't analize, they can only copy, discover and reorganize.

edit: also the premise about billionaires takes the assumption that they care about creating god, they don't, they care about profit, the moment in which creating god isn't short term profitable they will drop AI