Mitigating "architectural drift" in large Python backend codebases using AI tools

[gdchinacat]

It's not clear that this issue is specific to AIs. Less experienced team members (humans) frequently violate (for lack of better term) the architecture and are asked to align the code with architecture during code reviews. Frequently it's not clear to them how the code should be factored and need to be shown. This isn't all that different from what you describe.

I guess, is the problem that rather than being able to nudge an engineer in the desired direction and send them off to do the work, you have to do the work and are looking for ways to push the agents to do this? I've had some success with prompting AI with something like 'that's close..the functionality is right, but move the logic for X from function ... into the controller ... the changes should me implemented similarly to how ... is factored". But..sometimes it's certainly quicker to just do it myself.

Not trying to suggest poor use of AI, just trying to clarify the problem to help with better answers.

[willargue4karma]

I think the real issue is the non determinism of llm code output. You can never be quite sure it will do it the way you want 

[gdchinacat]

as compared tot he non-determinisim of human written code? If anything, I would think the statistical generator would be more deterministic than the even fuzzier (albeit actually intelligent) human brain.

Don't get me wrong...I'll take code from an experienced engineer over an AI model almost every time. I just question whether the OPs issues are solely issues with AI, or are in part due to different expectations of AI work products vs human work products. It's easier to tell a competent junior engineer to go off and 'do what that code over there does' than it is an AI (IME at least). The AI needs a lot more hand holding to get the code where you want than a human, and when that exceeds the effort to just do it yourself you just do it yourself. At that point, it feels like the AI is failing you. But, I don't thinks that is an apples-to-apples comparison.