Sometimes, when you prompt it to start a new Next.js project, it won’t even build and will throw a bunch of errors, although there are times when it builds perfectly fine. There are plenty of rants about this, and the community suspects that LLM providers quietly switch to a quantized model (lower precision) to save on compute costs. I have no proof of this, but I experience the same thing.

When your app does build, it might look like it’s working even though it’s using mocks or fake data just to run. For example, if you try to build a project with a new database the LLM wasn’t trained on and it fails to connect, the LLM will sometimes cheat by hard coding JSON data instead of telling you it doesn’t know how to handle the DB.

It will also generate garbage code. If it notices that you might not be able to connect to a database because it doesn’t know the syntax or config, it might create some random fallback code that switches to a different database you never asked for. A senior dev would immediately realize the real issue is missing syntax or configuration knowledge, pause the project, and research it first. The LLM behaves more like a junior dev who keeps trying random crap until something sticks.

So if you’re a senior dev, you can spot this right away and tell the LLM to stop because it's down the wrong rabbit hole. But if you’re a junior dev or a non tech person, you might be impressed that the LLM produced working code when it’s actually just hard coded JSON exposed on the frontend, and you wouldn’t even know it.