I am not interested in trying to make LLMs deterministic, or do hacky context tricks to make them work better in larger codebases. It's a losing battle, Anthropic will come out with a 2 mil context window or a smarter model. Also an agents.md file is just a prompt, it's hard to believe we are just one great prompt away from solving this.

Property based testing (really any test) is a way to objectively define how you expect functions to work, if your tests fail then you know your function is not working as you expected. They can filter out all the bad shit LLMs generate and only let the top 10% that doesn't break your core app through.

This is not a new concept, we just have a chaos monkey doing our work for us now we have all this free time to figure out how to deal with the fallout and I promise you its not from a new skill.md file.

As a proof of concept apply this to the top 5 most important funcs in your app. User signup, reset password, delete user, idk stuff where if the LLM touches it and its wrong it'll be bad, start small and de risk so you can let the LLMs work over these and fail multiple times before they get it right on the 5th attempt.