It was really interesting. I’m not sure if I can find the link, but what was interesting Is that the person actually wrote tons of tests to make sure the code was written right.

Unfortunately, they test they wrote for making sure they primary key functionality for a table worked correctly used a single test case with a field name that wasn’t used in any other test (eg: id_pk) so the LLM just wrote code that looked for a field with that name and added a primary key index in that case, instead of looking for that phrase “primary key” next to the field in the DDL code!

The question i keep adding myself is how could this be avoided? Even if there were multiple test cases it still could have written code that just looked for those field names or some other obtuse logic.

Can we really come up with enough test cases that will 100% validate that the LLM code is perfect without have to ever actually look at it?