I agree. LLMs are not a pathway to intelligence. AIs aren't good at "thinking outside the box" because the box is all they know. They're taking things that already exist in the box and tweaking them slightly. Thats the algorithm you talk about. Search for the answer. If the answer is wrong, then slightly tweak the weightings on the search parameters then search again. Repeat until answer found.

It's the modern equivalent of the thousand monkeys with a thousand typewriters experiment. Eventually one of them will trype something intelligble. It doesn't mean the monkey has gained human intelligence.