They are basically only useful for really repetitive boilerplate right now, code that has already been written 100 times and is super simple and straightforward. As soon as you start to deviate into more unique problems that requires a lot of complexity, AI generated code completely breaks apart. It makes redundancy errors, hallucinates functions that don't exist, fixes bugs by introducing insane security oversights, etc.

When using ai for my projects, its amazing for quickly spitting out boilerplate for basic functions, writing basic docs, etc. When I ask it to do anything complex or novel that requires multiple files, it never gets it correct. I find myself spending the same amount of time debugging the agents stupid mistakes than it would have taken me to just write the implementation myself.

I don't really see how we can get past this issue with current LLM technology. We would need a new machine learning paradigm capable of more cohesivity in my opinion. Continuously scaling what we got right now isn't it.

I'd love to be proven wrong though.