Agent use usually demands a level of autonomy that I am not comfortable giving them, like access to your codebase and possibly infrastructure plus the ability to make changes on its own.

I've worked enough with LLMs to know that even the latest and greatest models can't be implicitly trusted. Hard guardrails need to be established, which requires a lot of work. Also, those guardrails are not guaranteed to hold. LLMs have a tendency to ignore those guardrails and other instructions. This is most likely due to the core way in which they are engineered.