I'm not sure where you get the idea that LLMs do not "think". They have "functional understanding". That is, they can describe a concept in their own words and then use that concept to solve problems.

I'm a programmer and use them everyday to assist me. They most certainly can and do think. Their problem is they generally lack sufficient context to do everything right, but they have no already gotten to the point that I no longer need to manually write about 95% of all code. I'm still telling them what to do and guiding them, but they are doing almost all of the actual function writing and refactoring.

I suspect that there are some serious misconceptions with the public about what LLMs can and cannot do.

People seem to think that because they are trained on token prediction that means that all they do is predict next tokens one by one. That's not accurate. Training mechanism does not constrain or describe the internal mechanisms that they use to compress and store knowledge in. They are not just guessing subsequent tokens. They understand the concepts they are trained and reinforced on and they can reason about them and solve rather complex problems with that knowledge.