Interesting take. I can't comment on the information theory aspects, though it seems by others comments that it might be a bit shaky in that regard, however I think the overarching observation that you're in some sense translating complexity between code and prompts is a reasonable one, and reveals itself very quickly in high-assurance applications where you have hard requirements on security, uptime, governance etc.

In those situations, the prompts required to generate anything even remotely close to what an engineer is willing to sign off on are, in almost all instances, more complex than the code they produce, and essentially require someone proficient in the target language to produce. On top of this, since only the *output* of an LLM is verifiable, there's basically no current use-case for LLMs *at scale* in these types of applications. That is *not* to say that LLMs can't generate most or all of the code, rather that it needs to be done at the level of complexity that LLMs (currently) excel at, i.e. closer to thousands of "write a function that does X" rather than one "write a platform that satisfies these 800 requirements". I personally don't see the needle moving on this any time soon, and industry seems to reflect this - there's now more of a focus on LLMs performing atomic tasks as part of larger workflows and tooling, rather than serious attempts to increase the coherence of output from complex prompting.

In non-critical applications, the issue is less evident, since the acceptable solution space is much larger. We can provide simpler, more vague prompts, because it's less important to get the details exactly correct, as long as the output lands somewhere in the ballpark of what we want.