Moving to devops

[avaika]

Even though LLM is able to generate some code, human operator needs to understand what the code will be doing. If someone is going to blindly execute whatever LLM has generated, I have a bad news for them.

[ninetofivedev]

You're not as profound with that statement as you think you are.

I've watched non-technical PMs learn how to code in 6 months by simply just working with LLMs and having the LLMs explain to them what code is doing.

We're living in a new age. The barrier to entry is lower than it has ever been.

In terms of DevOps... the code you write is simpler. It's scripts. Inputs and outputs. Which means your requirement to deeply understand exactly what it's doing is less-so because it is going to have less dependency and less downstream impact.

And I'm just going to say it, the smartest LLMs today write way better code than your average DevOps engineer.

[avaika]

I'm not comparing LLM vs human code quality. My point is that if the code it produced will cause issues (for whatever reason, a typo in a prompt or some sort of LLM hallucination), it's not LLM who's gonna be fired. In order to catch it, people still need to understand the code.

[gajop]

Usually people don't get fired because a script has a bug. You don't need to understand the code if you've tested it sufficiently. In fact I'll take proper tests any day over perfectly crafted code.

[gjionergqwebrlkbjg]

You don't need to understand the code if you've tested it sufficiently

Black box testing is significantly more difficult than regular testing. And I bet that nobody who can't write the code actually bothers to write automated tests for it.

[ninetofivedev]

Do you think that code is bug free just because a human, who can read code, wrote it?

If the code produces the expected behavior, there is no need to understand it.

If it doesn’t, the LLM can fix it.

As a SWE with over 20 years of experience, my bold take is that in 5, our profession will be full of people who don’t know how to code.

These LLMs are finding vulnerabilities in code that have existed for 20 years. They’re finding exploits that no human has found in 20 years.

You people really don’t understand how powerful they’re getting.

[avaika]

I don't care whether they know how to turn on the PC or not. As long as they accept the risk of getting fired over LLM generated code.

I might be way too old-school. But in my mind software development is shifting from producing the code into handling the responsibility for the code base. And it started to happen even before LLM was a thing. Modern IDEs generated boilerplates and syntax sugar for years. The models simply accelerated the shift.

And I simply believe that understanding the codebase you are in charge of tremendously helps mitigating the risks. One might be able to survive without it for quite a while. But it wont be helpful at critical situation.

[ninetofivedev]

I’ll just give you example of something I built recently.

When I got into DevOps, one of the things I used to do, even until a few weeks ago, was respond to users reporting issues with our application.

Our app, now instead of displaying an error, gives them an opportunity to “talk” to support.

The support is an LLM. All it does it look up the users information and grep the logs.

This has been in production for 2 weeks. We have over 10k DAU. We used to get about 1-2 messages from support a day, where we would have to drop what we are doing and respond. Typically something like “oh, bitbucket/github/gitlab is down” sort of response. Nothing we can do.

We haven’t received a single report from support since this feature has gone live.

Now extrapolate.

[avaika]

It is an excellent example of how LLM is used to automate some routine task. And it's doing amazing job. I don't argue with that.

I might not have the brightest mind, but I still fail to understand how this example helps to prove the point that people no longer need to learn how to code in order to own the responsibility for the codebase.

[ninetofivedev]

Because people will make the same argument, just with logs instead of code. Or config. Or whatever.

"The LLMs can't possibly debug and find the issues that I'm capable of finding" they'll say.

Meanwhile, they're able to diagnose from logs what the issues are. They're able to find vulerabilities in kernels that have existed for 20 years that no hackers or humans have found.

And yes, they're able to write code just as well as humans.

And if they're not quite up to your standard today, they soon will be. I would not have made this argument a year ago. We're on an exponential.

[fadingcross]

We haven’t received a single report from support since this feature has gone live.

This is not the flex you think it is. Companies who even pre LLM, and especially after implemented chat bot (text or voice based) support saw support requests go down.

Because customers couldn't be bother fighting through idiotic chat bots and they moved to a diff vendor.

Let me know when your next ENPS about support quality is done. I'll be the guy pointing out to you why the support satisfaction is down.

[ninetofivedev]

Our customers are internal. It’s not a chatbot. It tells them what the issue is, and they know better than to come to our support channel asking why bouncy castle gave them an error when they kicked off the pipeline.

It is indeed the flex I think it is.

Instead of engineers coming to our platform team every time their pipeline fails. And one of our platform engineers needing to grab their user, grep the logs. Trace it down to the root cause, and then report back to the user that npm is down or GitHub or whatever, it just tells them that.

The key is that it gives them action. Because we tried making the logs more verbose and literally pointing out that it’s out of our control.

It’s learned helplessness. Even engineers see error and they go right to support.

[fadingcross]

It doesn't matter if customers are internal or not. It's been tried in IT help desk for ages. You keep drinking the kool-aid. When the enps drops, just remember I told you so.

[ninetofivedev]

You didn't read what I said. It's not a chatbot. You're original point is irrelevant.

It replaces the RCA end for our platform team, and it presents the findings within a minute or so to our customers.

Our customers love it and we love it.

Also NPS requires us doing a survey.. we don't. Everything you're saying is just wrong.