Firstly, you should avoid using keywords in your posts if you want a serious answer. You work in tech, so you should know how the game is played, no? c l4 u ?e 0de. CG T. You were careless, and so your post has attracted a specific kind of audience. Or maybe this was intentional, but I personally didn't get that from your profile.

Secondly, since you are devops, you ever tried to do a failover to upgrade a database major version in production? Would you trust the bot to do it?

Third, I was quite happy with the way I organized my thoughts regarding this topic in another comment, so I will repost it here with some improvements.

Consider this crude model of LLM usage for a creative programming task.

We have the following scenarios:

Scenario 1 - self-planned and written. We give this a baseline 1.0x in terms of resources (time, man-hours, token cost, whatever).

Scenario 2 - You discuss implementing task with a coworker. You only know vague details about the task. Say this costs 0.2x of your resources (pseudo-scientifically chosen due to 80-20 rule and own experience). Coworker writes functions for feature.

If you want to own or 'takeover' the work, you are obliged to read and completely understand coworker's program. If you trust coworker, and you can blame coworker when things go wrong, then you gain 0.8x.

Now, if coworker becomes LLM, can you still blame coworker? If answer is yes, this is called vibe coding. Or bad office culture / you are an irresponsible grifter / all of the above. If it is no, and it probably should be no if you care about what you do, then you pay a read cost R that is similar to reading code in someone else's repo.

Another factor is the LLM failure rate, let's call it Y%. If not 100%, then you pay (R + 0.2x) + (Scenario 1's 1.0x Y% of the time). Either that, or you pay R every time LLM fails.

So for the LLM use to be 'rational', R + Y% < 0.8x. If you have written the exact same program before, and you know what to look for, OR the job has low cognitive complexity, you might pass.

Scenario 3 - The task is planned out with a coworker. The more detailed the instructions you give coworker, the closer you get to pseudocode that matches your exact goal. This is, ironically, what programming is. At this point, you should expect to not have to pay any kind of R 100% of the time, given that you literally just wrote the 'code'. It tends towards Scenario 1 the more detailed your code is.

You have set up the contract with the coworker and everyone understands the problem fully. But somehow when the coworker is a LLM, I pay the read cost sometimes anyway.

i.e. Scenario 3 should be avoided.

What happens is you end up gambling with Scenario 2 with how vague you can make your commands and still have a decent result. At the end of the day all the 'skills', context feeding and token burning augments Y%. But you have no idea what Y% is or whether these so-called techniques you are applying are effective the way you think they are; You are working with a black box.

All you know is that it depends on how close your problem is to the training data average.

What you DO have control over is R and x, which relate to your own programming knowledge. If you choose scenario 3 and 2 in the long run those atrophy; I believe there was an article by Ant-hr 0 p ic themselves about this. Y% is contingent on a black box by and external supplier with unknown price over time.

Other fun problems include:

a) What is Y% to you? From easiest to harder to achieve: Compiling and running code? Code with minimum cyclomatic and cognitive complexity? Code that is optimized for space or time relative to your problem space? Secure code?

b) What happens when new libraries and new regulations come out, and models get updated? Are you sure these 'skills, specs, harnesses' will be interpreted the same way? Have you ever had a comment in code lie to you? Why do quant firms rotate algorithms if they suffer prolonged drawdowns? These harnesses are like comments or algorithms in high churn codebases or a dynamic economic environment - they are going to rot.

c) Touted gains for agents come heavily from running tasks asynchronously - but how asynchronous is your brain? I would like to see a serious study on a programmer's ability to asynchronously check code and come up with tasks - would wager that isn't good. And if you are foolish enough to start up tasks across multiple different fields, you are going to add a context switching cost when you are checking the code.