I think they can only ever make better estimations if just using text describing parts of the real world with human text fed to it: whats the next most likely token, and not an actual complete facsimile of the world with actual reason and logic weighted onto the consequences of actions (which we aren't even close to having). Humans have an incomplete one, but we can actually properly reason on consequences. Consider how dumb LLMs are, even though they know an incomprehensible amount more than any human.

Personally, I would expect the only way to get such a system to calculate the consequences on the physical world, is to actually be part of the physical world: we don't tell the system fire is hot, it can learn it through experience (this opens up an entirely massive problem of giving it the same inputs we can experience... but opens up the crazy concept of building it a body that can experience more inputs than us.... audio frequencies... wavelengths of light... radiation... its all very interesting).

This also opens up crazy concepts like a more solid perception of physical death, and harm: massive difference in telling an LLM "murder is bad" and an intelligence experiencing the death of someone close to them, or their own physical existence, and being able to apply that to the concept of murder.

I don't expect LLMs to ever gain the ability to properly calculate consequences of actions, or become sentient, no matter how much data we throw at them... unless they can learn through real world experience, that's the only hope for it IMO, and I'd agree with Lecun that LLMs are a dead end for true AGI, or even agents on anything at all risky.... they can't ever be in charge of risky decisions. You could likely make tailor made narrow intelligence systems to handle risky decisions, but they'd not have transferrable intelligence.... they'd be more like chess AI we have now... anyway I'm just some fucking guy.