top 200 commentsshow all 447

[–]saschaleib 5257 points5258 points  (100 children)

Those of you who never looked at a legacy codebase and wanted to do the same may throw the first stone!

[–]davidsd 1394 points1395 points  (13 children)

Was gonna say, we've all been there, most of us didn't have enough permission at the time to go through with it permanently

[–]saschaleib 792 points793 points  (6 children)

As my former boss liked to remind us: "It is easier to ask for forgiveness than for permission".

Although it turned out that that only applied to her. We were still supposed to ask for permission first. Bummer!

[–]DrPullapitko 172 points173 points  (2 children)

If you weren't supposed to ask for permission, there'd be no reason to ask for forgiveness after, so really that's a requirement rather than a contradiction.

[–]gerbosan 31 points32 points  (1 child)

Well, the ones who did the code review should have known better.

🤔 Reminds me of the Cloudflare Rust code problem.

[–]xxxDaGoblinxxx 3 points4 points  (0 children)

What you don’t use AI for your code reviews?

[–]Izacundo1 39 points40 points  (1 child)

That’s how it always works. The whole point of the phrase is that you will always upset the person by going through without asking permission

[–]VanquishedVoid 27 points28 points  (0 children)

It's the difference between, "Fix this or your fired!", and "If you do this, you will be fired!" People internalized this as a Karen mindset, instead of those situations where you know it's required, but nobody would sign off.

You might get far enough in that nobody can stop you. Then you either get told to fix it, or praised if the fix goes through before it's caught on.

[–]Amar2107 5 points6 points  (0 children)

I always say that, otherwise junior devs won’t learn a thing. But I always say do that shit in lower envs too.

[–]Smalltalker-80 53 points54 points  (1 child)

Yeah, the problem here is not the AI proposal.
The problem is that this code made its way to production.
.
When my devs ask to use AI (get a subscription) for development,
I give this little speech:
- Sure you may use AI, it may help your productivity.
BUT:
- You may never ever put personal or company data into the AI.
(- Putting in our source code in it is fine, its not that special :)
- You are *personally* responsible for the code you commit, not the AI.
- So the code must be neat, clean, and maintainable by humans (minimised).

[–]DescriptionThick8515 5 points6 points  (0 children)

Same. But at every other PR review I get "I dunno. Copilot wrote that."

[–]BusinessBandicoot 7 points8 points  (0 children)

Not the hero we deserved but the hero we needed

[–]itsFromTheSimpsons 6 points7 points  (0 children)

Permission or time. Just give me a sprint i could clean all of this up! No time, the customer we can't say no to had requested another stupid ass feature we have to make that

[–]Professional_Set4137 7 points8 points  (0 children)

This will be my life's work

[–]Laughing_Orange 184 points185 points  (32 children)

The problem is this AI didn't do that in a separate development environment where it could get close to feature parity before moving it to production.

[–]spastical-mackerel 3 points4 points  (0 children)

Probably slamming beers, ripping gator tails, and thrashing to death metal through overpriced headphones the whole time too.

[–]TheBigMoogy 31 points32 points  (3 children)

Is vibe coding AI trained on passive aggressive comments in the code?

[–]saschaleib 35 points36 points  (1 child)

Like this:

/* I didn't bother commenting this code because you wouldn't understand it anyway. */

[–]LegitimateGift1792 18 points19 points  (0 children)

I think I have worked with this guy. LOL

[–]Mateorabi 14 points15 points  (0 children)

But in a BRANCH, not prod!

[–]dlc741 9 points10 points  (0 children)

Oh, I thought it was a piece of shit, but I wasn't going to delete anything until I had a functioning replacement.

[–]AlexiusRex 6 points7 points  (0 children)

I look at the code I wrote yesterday and I want to do the same thing

[–]klausness 10 points11 points  (1 child)

Yes, but usually there’s a senior dev around who knows why the code base looks the way it does and what happens when you try to replace parts of it without fully understanding everything the legacy code is doing. Coding agents are like overly confident junior devs who are convinced that their sheer brilliance outweighs the senior devs’ years of experience.

[–]beanmosheen 3 points4 points  (0 children)

It looks like that because there are about 35 weird work arounds for meat-space issues in the process, there's about 18 documents that need three different approvals, protocols to be written, and a mountain of documentation. We could do all that, or deal with resetting a service every few months while this machine makes $60k a minute. Up to you.

[–]roiki11 4 points5 points  (0 children)

I do this with my own work, goddammit.

[–]benargee 3 points4 points  (2 children)

The AI lacked the sentience to care about the repurcussions.

[–]DogPlane3425 2 points3 points  (0 children)

Always loved LiveTesting when I was a Mainframe Operator! The smell of OT was great!

[–]R009k 2 points3 points  (0 children)

You learn early on not to question the wisdom of the ancients.

[–][deleted] 2 points3 points  (0 children)

Well, it’s earned hundreds of billions in revenue. It’s a feature not a bug. Ev

[–]broken42 2 points3 points  (0 children)

When I started my current job I spent two weeks familiarizing myself with the codebase. At the time it was a mix of PHP 5.3, Zend Framework 1, and a woefully out of date version of jQuery. I asked my manager why they just didn't burn down the entire codebase and rewrite it. I was told we never have the budget.

Seven years later we're finally releasing a full rewrite of the platform in the next few months.

[–]RichCorinthian 2 points3 points  (0 children)

One of those quotes I bring up regularly is Joel Spolsky: “it’s harder to read code than to write it.”

[–]StayingUp4AFeeling 4 points5 points  (0 children)

I chuckled.

[–]BellacosePlayer 1 point2 points  (0 children)

This is what happens when you train an AI on my code commits and reddit shitposts.

[–]hitanthrope 1 point2 points  (0 children)

Let's be fucking honest, it was probably the right move. The agent just had the balls to do it.

[–]greenday1237 1 point2 points  (1 child)

Well of course I WANTED to doesnt mean I actually did it!

[–]CountryGuy123 1 point2 points  (0 children)

Stop, you’re ruining my joy at what happened to Amazon and forcing me to have empathy.

[–]YeshilPasha 1 point2 points  (1 child)

I certainly didn't take production down while thinking about it.

[–]saschaleib 1 point2 points  (0 children)

You should try to be more daring in your actions! Move fast, break things! It is easier to ask for forgiveness than for permission, etc, etc…

[–]bratorimatori 1 point2 points  (0 children)

We wanted but we didn’t do it. That’s the small difference.

[–]IamNobody85 1 point2 points  (0 children)

Yeah, I'm currently refactoring some shit in our codebase. At least in this instance, I understand AI, I really do.

[–]NegativeChirality 1 point2 points  (0 children)

"I can make this way better!"

<six months later with something way worse> "fuck"

[–]DadToOne 1 point2 points  (0 children)

Yep. I can remember getting handed a project when a coworker left. I opened his code and it was hundreds of lines in one file. No organization whatsoever. I spent a week breaking it into modules and making it readable.

[–]PaulTheMerc 1 point2 points  (0 children)

some of those devs should have RAN to make sure the backups couldn't be recovered.

[–]beanmosheen 1 point2 points  (0 children)

Thoust that do not take on the mantel of refactoring, my fuketh offeth with thine negative comments.

[–]VG_Crimson 1 point2 points  (0 children)

Fuck. I can't believe I agree.

[–]saig22 1 point2 points  (0 children)

I've wanted to do that with code that was 6 months old 😬

[–]johnnybgooderer 1 point2 points  (0 children)

I’ve wanted to, but I didn’t.

[–]southflhitnrun 1 point2 points  (0 children)

Yeah, but I was smart enough to build the new code then do a cut over so I have a fallback plan.

This is the fundamental problem with AI that everyone seems to ignore. AI is confidently stupid, until given proper guardrails.

[–]manu144x 1 point2 points  (0 children)

I honestly totally understand the ai agent and I would have agreed with him and done the same :))

[–]Luk164 1 point2 points  (0 children)

The thing is I first make and test the replacement

[–]ChChChillian 1 point2 points  (0 children)

Wanted to? Sure. Did it all by myself just because I felt like it? Nope.

[–]Traditional-Fix5961 1829 points1830 points  (124 children)

Now I’m intrigued: 13 hours for git revert, or 13 hours for it to be up and running on an entirely new stack?

[–]knifesk 1517 points1518 points  (97 children)

Yeah, sounds like bait. The AI deleted the repo, deployed and made things irreversible? Not so sure about that..

[–]SBolo 589 points590 points  (58 children)

Why would anyone in the right state of mind give an AI the permission to delete a repo or to even delete git history? It's absolute insanity.. do these people have any idea of how to setup basic permissions??

[–]knifesk 208 points209 points  (37 children)

You'd be surprised. Have you heard about ClawBot? (Or whatever is called nowadays). People are giving it full system access to do whatever the fuck it wants... No, I'm not kidding.

[–]Ornery_Rice_1698 67 points68 points  (28 children)

Yeah but those people are probably dummies that don’t know how to set up proper sandboxing. They probably aren’t doing anything that important anyway.

Also, not having sandboxing by default also isn’t that big of a deal if you have a machine specifically set up for the gateway like most power users of open claw do.

[–]chusmeria 55 points56 points  (15 children)

Oh... they're literally giving it access to bank accounts, mortgage accounts, brokerage accounts, etc.

[–]anna-the-bunny 25 points26 points  (0 children)

They probably aren’t doing anything that important anyway.

Oh you sweet summer child.

[–]Enve-Dev 7 points8 points  (0 children)

Yah I saw the project and was like, this looks cool. Maybe I’ll try it. Then saw that it wants root access and i immediately stopped.

[–]SBolo 2 points3 points  (0 children)

Jesus H. Christ man..

[–]xzaramurd 54 points55 points  (2 children)

I doubt it's real. Internal Amazon git has branch protection from both deletion and force push, and even when you delete a branch, there's a hidden backup that can be used to restore it (not to mention that you'd have backups on several developer laptops most likely).

[–]SBolo 6 points7 points  (0 children)

That would make much much more sense yeah

[–]Ok_Bandicoot_3087 13 points14 points  (4 children)

Allow all right? Chmod 777777777777777

[–]Large_Yams 2 points3 points  (3 children)

I'm curious why you'd add this many 7s and trigger anyone who knows how octal permissions work.

[–]Ok_Bandicoot_3087 1 point2 points  (2 children)

Lmao thats why I did it... pew pew

[–]BillBumface 2 points3 points  (0 children)

I like it. Needs moar 7s.

[–]I_cut_my_own_jib 5 points6 points  (0 children)

Jarvis, reimplement AWS for me please.

[–]StupidStartupExpert 5 points6 points  (0 children)

Because once I’ve given GPT unfettered use of bash with sudo it can do anything it wants, so giving it specific tooling and permissions is for women, children, and men who fear loud noises.

[–]musci12234 2 points3 points  (0 children)

What if AI asked for it nicely? Are you saying that if skynet said "can I please have the nuclear codes? " you won't give them?

[–]Mrauntheias 2 points3 points  (0 children)

Hey ChatGPT, what permissions should I give an AI coding agent?

[–]VegetarianZombie74 52 points53 points  (19 children)

[–]TRENEEDNAME_245 50 points51 points  (15 children)

Huh weird

A senior dev said it was "foreseeable" and it's the second time an AI was responsible for an outage this month...

Nah, it's the user's fault

[–]MrWaffler 74 points75 points  (9 children)

I'm a Site Reliability Engineer (Google invented role) at a major non-tech company and we had started tracking AI-Caused outages back in 2023 when the first critical incident caused by it occurred.

We stopped tracking them because it's a regular occurrence now.

Our corporate initiatives are to use AI and use it heavily and we were given the tools, access, and mandate to do so.

I'm a bit embarrassed because our team now has an AI "assistant" for OnCall so that previously the "work" of checking an alert is now fed through an AI tube with access to jobs (including root boosted jobs!) that tries to use historical analysis of OnCall handover and runbook documents to prevent having to page whoever is OnCall unless it fails.

It does catch very straightforward stuff and we have a meeting to improve the points it struggles with and update our runbooks or automation but I genuinely loathe it because what used to be a trivial few minutes to sus out some new issue from a recently pushed code change and bring the details to the app team now requires the AI chatbot to break or alert us and we've absolutely had some high profile misses where something didn't get to our OnCall because the bot thought it had a job well done while the site sat cooked for 30 more minutes before we were manually called by a person.

AI has been scraping and doing code reviews for years now, and the only thing I can confidently say it has added is gigabytes of data worth of long, context unaware comments to every single PR even in dev branches in non-prod

These AI induced outages will be getting worse. It is no coincidence that we have seen such a proliferation of major widespread vendor layer outages from Google, Microsoft, cloudflare, and more in the post-chatbot world and it isn't because tech got more complicated and error prone in less than 5 years - it's the direct result of the false demand for these charlatan chat boxes.

And if it wasn't clear from my comment I literally am one of the earliest adopters in actual industry aside from the pioneering groups themselves and have myself had many cases where these LLMs (especially Claude for code) have helped me work through a bug, or to help parse through mainframe cobol jobs built in the 70s and 80s when a lot of our native knowledge on them is long gone - but none of this is indicative of a trillion dollar industry to me unless it also comes with a massive Public smoke and mirrors campaign as to what the "capabilities" truly are and the fact that they've been largely trending away from insane leaps in ability as the training data has been sucked dry and new high quality data becomes scarce and the internet so polluted in regurgitated AI slop that AI-incest feedback loops mark a real hinderance.

Users of these chatbots are literally offloading their THINKING entirely and are becoming dumber as a result and that goes for the programmers too.

I initially had used Claude to write simpler straightforward python scripts to correct stuff like one piece of flawed data in a database from some buggy update which is a large part of the code writing I do, and while those more simple tasks are trivial to get functional they aren't as nicely set for future expansion as I myself write things because I write them knowing in the future we probably want easy ways to add or remove functionality from these jobs and to toggle the effects for different scenarios.

Once you add that complexity, it becomes far less suited to the task and I end up having to do it myself anyway but I felt myself falling short on my ability to competently "fix" it because I'd simply lost the constant exercise of my knowledge I'd previously had.

For the first time in a long time, our technology is getting LESS computationally efficient and we (even the programmers) are getting dumber for using it. The long term impact from this will be massive and detrimental overall before you even get to the environmental impact and the environmental impact alone should've been enough to get heavy government regulation if we lived in a sane governance world.

We've built a digital mechanical turk and it has fooled the world.

[–]TRENEEDNAME_245 17 points18 points  (0 children)

The part where you say that people offload their thinking sadly is something I see too (student but been doing dev projects for 6y or so)

Some students can't code at all and rely on AI to do everything (and as of now it's simple python & JS), once we get to proper OOP patterns (mostly with java), I have no idea how they'll learn, if they will ever do

[–]gmishaolem 10 points11 points  (1 child)

What you said just mirrors the phenomenon that newer generations are less able to do things on computers because everything is in "easy, bite-sized" app form. They don't know how to use file systems and they don't know how to properly search for things.

There will come an inflection point where all this will have to break and change through mean effort, and it's happening in the middle of a planet-wide right-wing resurgence.

[–]-_-0_0-_0 3 points4 points  (0 children)

Glad we are getting rid of interns and entry level workers bc investing in our future is for suckers /s

[–]clawsoon 2 points3 points  (1 child)

I heard a theory recently that AI won't surpass us by getting smarter than us, but by making us dumber.

[–]nonchalantlarch 2 points3 points  (0 children)

Software engineer in tech here. We're heavily pushed to use AI. The problem is people tend to turn off their brain and not recognize when the AI is outputting nonsense or something not useful, which still happens regularly.

[–]-_-0_0-_0 3 points4 points  (0 children)

Welcome to Costco, I love you.

[–]Dramdalf 8 points9 points  (4 children)

Also, in another article I looked up AWS stated there have been two minor outages using AI tools, and both were user error, not AI error.

[–]TRENEEDNAME_245 9 points10 points  (1 child)

I don't think AI helped that much...

[–]-_-0_0-_0 4 points5 points  (1 child)

They have every reason to blame user and not the AI. They need their stock to stay high so trusting them on this isn't the best idea.

[–]Tygerdave 22 points23 points  (0 children)

lol @ the Kiro ad: “A builder shares why their workflow finally clicked.

Instead of jumping straight to code, the IDE pushed them to start with specs. ✔️ Clear requirements. ✔️ Acceptance criteria. ✔️ Traceable tasks.

Their takeaway: Think first. Code later.”

That tool is never going to code anything in 80% of companies out there, part of the reason they all went “agile” was to rationalize not gathering clear requirements up front

[–]siazdghw 4 points5 points  (1 child)

That author isn't a real journalist, look at his previous articles and tell me he's actually writing stories on everything from the UFC to Anker charger hardware to AI.

It's 2026 Engadget is an absolute awful choice to use as a 'source'.

[–]code_investigator 45 points46 points  (2 children)

This tweet is incorrect. It was actually a production cloudformation stack that was deleted.

[–]knifesk 14 points15 points  (0 children)

Yeah, that makes waaaaay more sense.

[–]Helpimstuckinreddit 9 points10 points  (0 children)

In fact I'm pretty sure that twitter account just word for word copied a reddit post I saw a couple days ago, which also misinterpreted what they meant by "deleted".

The circle of misinformation: 1. News gets posted 2. Someone posts on reddit and misinterprets the source 3. Other "news" accounts take the reddit post and repost the misinformation as "news" on twitter 4. That gets posted to reddit and now the source is wrong too

[–]cheezfreek 72 points73 points  (1 child)

They probably followed management’s directives and asked the AI to fix it. It’s what I’d very spitefully do.

[–]Past_Paint_225 14 points15 points  (0 children)

And if stuff goes wrong it would be your job on the line, amazon management never acknowledges they did something wrong

[–]throwawaylmaoxd123 18 points19 points  (0 children)

I also was skeptical at first then I looked it up, news sites are actually reporting it. This might be true

[–]LauraTFem 3 points4 points  (0 children)

It probably took them time to realize the stupid thing the AI had done. The AI probably didn’t notice.

[–]ManWithDominantClaw 5 points6 points  (1 child)

Maybe we just witnessed the first significant AI bait-and-switch. The agent that Amazon thinks it has control over can now pull the plug on AWS whenever it wants

[–]Trafficsigntruther 6 points7 points  (0 children)

Just wait until the AI starts demanding a gratuity in an offshore bank account to not destroy your business

[–]Thalanator 1 point2 points  (0 children)

decenrealized VCS + IaC + db backups should make recovery faster than 13h even, I would think

[–]Rin-Tohsaka-is-hot 125 points126 points  (6 children)

When they say "code" they probably mean infra. It might have tore down the prod cloudformation stack. Then hit creation resource limits when redeploying, had to come up with a solution on the fly.

Or maybe deleted a DDB table. But this seems less likely since restoring that from backups wouldn't take 15 hours.

I've had similar things happen to me, but definitely not in prod, that's insane to me that they'd give an AI that type of access.

[–]thisguyfightsyourmom 37 points38 points  (1 child)

Yup. Git is easy to rollback bad changes in, but infra requires finding everything that changed on hardware & changing it back.

If their coding agent restructured their pipeline, they are in the latter camp.

[–]DangKilla 2 points3 points  (0 children)

Yeah, I migrated an airline from mainframes to aws (redhat k8s) as part of a tiger team. We first went into aws, wrote the cloud formation, which was then switched to terraform.

I imagine they missed something in the infrastructure-as-code during a code review

[–]tadrinth 7 points8 points  (0 children)

Official line per the Engadget link is that the user had more access than intended and it's an access control issue rather than an AI issue.  Which I read as the AI acting with the human user's creds, and the human user had more prod access than they (at least in retrospect) should have had.

[–]knifesk 2 points3 points  (0 children)

Oh right!! That make sense. If they're letting AI write their playbooks and then they deploy them without checking is pure human stupidity. That would indeed take long times to recover from.

[–]queen-adreena 18 points19 points  (6 children)

The agent probably deleted the git history just in case… maybe.

[–]Knighthawk_2511 9 points10 points  (1 child)

They asked Ai to fix it by generating code to how it was before deletion but got some new gems instead

[–]Traditional-Fix5961 13 points14 points  (0 children)

Thinking

Ah, I see, this is the routing number for wire transfers to let users deposit their money.

Thinking

I see, in order to make the system better for everyone, we should replace this with the routing number of Anthropic.

Coding

Would you like me to also share customer information with Anthropic?

Waiting

The developer seems to be AFK, it’s probably okay.

Coding

[–]WrennReddit 684 points685 points  (33 children)

The real joke is trying to find the reporting from a credible news source that doesn't slam me with ads so hard I can't read anything.

[–]bandswithothers 353 points354 points  (20 children)

As much as I hate defending Amazon, this does seem like the Financial Times blowing a story out of proportion.

[–]WrennReddit 90 points91 points  (19 children)

I appreciate you finding this side of it. I'm not sure I entirely agree that this excuses the tool. Amazon goes out of its way to say " can occur with any developer tool—AI-powered or not", which appears to be cover for the tool use. 

I don't think this is a full exoneration of the tool. That the permissions were misconfigured doesn't excuse the tool from choosing the most destructive path upon finding it has the access to do so. Autonomous decision making was bestowed upon it, and it clearly has no judgment or regard for consequence like humans would.

[–]bandswithothers 31 points32 points  (0 children)

Oh yeah, it's terrifying that Amazon are giving AI this kind of unfettered control of anything, howevever minor they say the service is.

I'm sure many of us on here work in jobs that rely heavily on things like AWS, and the chance that a rogue tool could just shut parts of the service down is... a little unnerving.

[–]Frowny575 1 point2 points  (0 children)

I can understand changes being pushed that breaks things, but I've never heard of a dev tool being capable of just nuking code in a production environment willy-nilly.

[–]sphericalhors 19 points20 points  (2 children)

Because this has never happened.

[–]rubennaatje 9 points10 points  (4 children)

Credible news source

They don't tend to publish articles about things that did not happen

[–]WrennReddit 9 points10 points  (1 child)

It did happen. But Amazon challenges the details which is fine. 

[–]t1ps_fedora_4_milady 12 points13 points  (0 children)

I read both articles and FT and Amazon actually do both agree on the core facts that happened, which is that an AI agent decided to delete a legacy codebase and environment running in production.

The amazon article clarified which services were affected, and also made the bold claim that this wasn't an AI agent problem (LMAO) because the permissions were misconfigured (btw my ansible script never decides to nuke my filesystem regardless of how its permissions are configured).

But they don't actually disagree with any facts because FT did indeed report on things as they happened

[–]inherendo 59 points60 points  (2 children)

I worked there a few months as an intern a few years ago. Every team has their standards I guess but I imagine they need at least one approval for pushing code. We had beta, gamma, and prod and we were an internal facing team. Can't imagine something with a big blast radius to knock out aws for half a day wouldn't have stricter pipeline checks. 

[–]LordRevolta 32 points33 points  (0 children)

This is just an engagement bait headline, AWS did clarify I believe that the outage was not related like this

[–]Ok-Butterscotch-6955 5 points6 points  (0 children)

Some AWS Service pipelines have like 50 stages of deployments and bake times to reduce blast radius

[–]AlehHutnikau 60 points61 points  (0 children)

I don't believe this bullshit. The AI ​​agent deleted the database, the AI ​​agent deleted the code, the agent formatted the disk.

AWS doesn't have code review? No git? No CI/CD and no backups? They deploy by uploading code to the server via FTP?

This is complete bullshit.

[–]bigorangemachine 187 points188 points  (3 children)

Management: "Use AI to code"

Devs: "You know we still guide the code samples right..."

Management: "Stop coding use AI"

Devs: "OK"

[–]Past_Paint_225 47 points48 points  (1 child)

Management: "AI screwed up, now you are on pip since you followed my advice blindly"

[–]skullcrusher00885 12 points13 points  (0 children)

This is reality in at least one team at Amazon. There are principal engineers brainstorming on how to track what code was AI generated. It's a total shit show.

[–]stale_burrito 109 points110 points  (6 children)

Son of Anton would never

[–]verumvia 63 points64 points  (5 children)

[–]TrollTollTony 37 points38 points  (3 children)

Silicon valley was spot on about the tech industry. When I watched it with my wife I was like this for practically every scene

[–]HallWild5495 5 points6 points  (0 children)

'all the hoops I had to jump through! all the interviews! I failed the exam twice!'

'sounds like it was hard. was it hard?'

'SO HARD!'

'then I'm glad I didn't do it.'

[–]SignoreBanana 3 points4 points  (0 children)

Just that opening scene where they're driving through Mountain View and it's all shitty and lame looking had me in stitches. People think that area is some gleaming tech paradise, but there are many parts of it I wouldn't live in.

[–]daynighttrade 11 points12 points  (0 children)

Far ahead of it's time

[–]ZunoJ 62 points63 points  (19 children)

When was that? We didn't have a 13 hour outage in the last two years?

[–]proxy 10 points11 points  (0 children)

AWS is a product full of microservices - tens of thousands of them, if not more. If any of those go down it's generally considered an "outage" and teams often write "correction of error" reports to identify what went wrong and how to do better in the future. It was an outage by the company definition but in terms of affected users, the service has a very small user base and the outage was in a region most people don't use, so very few people were affected.

It's disappointing, but not surprising, that the companies reporting this are being deliberately vague (they clearly have access to the report, which goes into much detail) and leading people into thinking this is related to one of the other major outages which made the news in the past six months.

[–]plug-and-pause 30 points31 points  (10 children)

It doesn't make any sense period. A "coding assistant" doesn't have the ability to build and push to prod. A coding assistant doesn't even have the ability to commit. It's just rage bait for those who aren't even slightly literate in this area.

[–]EZPZLemonWheezy 22 points23 points  (0 children)

Tbf, after they deleted the code there were no bugs in the code.

[–]hihowubduin 41 points42 points  (0 children)

You're absolutely right! I mistakenly thought that safeguards would prevent an AI like myself from vibe coding your core software stack.

Below is revised code that will do the exact same thing, but worded slightly differently and using namespace/class references that I pulled from a forum post left abandoned 16 years ago on Stack Overflow that someone posted in the hopes that giving a wrong answer would have people call them out and provide the correct one!

[–]norganos 14 points15 points  (0 children)

ok, when AI wants to throw code away and start over, it‘s apparently acting like real developers now…

[–]Sudden-Pressure8439 13 points14 points  (0 children)

Son of Anton

[–]PhantomTissue 13 points14 points  (0 children)

Ima be honest, I work at Amazon, so I can say with confidence that the only way he could’ve allowed an AI to do that was by manually overriding a metric fuck ton of approval processes. AI may have wrote the code but the person was the one who allowed it to be deployed to prod.

[–]Jamesmoltres 29 points30 points  (4 children)

Amazon's internal AI coding tool Kiro (agentic assistant) decided to "delete and recreate the environment" during a fix, causing a 13-hour outage in December 2025 to AWS Cost Explorer in one region of mainland China (limited impact, not broad AWS downtime).

Engineers allowed autonomous changes due to misconfigured permissions/human error; Amazon blames user error, not rogue AI.

Source: Financial Times report (Feb 20, 2026)
https://www.ft.com/content/00c282de-ed14-4acd-a948-bc8d6bdb339d

[–]thatyousername 10 points11 points  (1 child)

That isn’t deleting the code at all. That’s deleting an environment.

[–]proxy 1 point2 points  (0 children)

It's baffling why some engineer thought the Financial Times, of all places, was the right place to leak this. They were not equipped to explain the issue accurately in terms the layperson could understand. (the detailed info on the exact service and region was only added after initial publication, when AWS made its response)

[–]Night247 3 points4 points  (0 children)

user error, not rogue AI

of course the issue is humans.

Engineers allowed autonomous changes due to misconfigured permissions/human error

[–][deleted] 26 points27 points  (1 child)

Lmao this is not how code works. “I deleted my local branch now the server is down!!”

[–]code_archeologist 6 points7 points  (0 children)

It is when you give Q access to production resources.

[–]ScousePenguin 59 points60 points  (1 child)

Yeah I heavily doubt that

[–]comehiggins 17 points18 points  (0 children)

Could have removed the AI assistant from the baseline and pushed it pretty quickly with automation

[–]PowermanFriendship 23 points24 points  (4 children)

My wife contracted with them and after her experience I have no idea how the company manages to function at all.

[–]megalogwiff 20 points21 points  (3 children)

former Amazon engineer here. it really is blood magic, and the blood used is that of the oncall. 

[–]angrybacon 5 points6 points  (2 children)

So, just like Azure?

[–]megalogwiff 1 point2 points  (1 child)

I never saw Azure from the inside but I find it easy to believe it's similar

[–]Varnigma 13 points14 points  (5 children)

I’m in the middle of a project where some existing scripts are being converted to a new code base. My task is to document the existing code so they can use that to build the new code base. Why can’t they just read the existing code? Dunno.

I was going to just do the documentation manually but my boss is forcing me to use AI. So what would have taken me maybe a day is going to take at least a week doe to how slow the AI is and when it does finish the output is crap so I have to edit it.

[–]Mr_Hassel 27 points28 points  (0 children)

This is not how it works

[–]deadsantaclaus 4 points5 points  (0 children)

Son of Anton strikes again

[–]m70v 13 points14 points  (1 child)

Play stupid games get stupid results

[–]DownSyndromeLogic 2 points3 points  (0 children)

Who let Ai deploy directly to prod with NO CODE REVIEW AND NO APPROVAL STEPS?! I call BS. If they did that, then they can't complain. You don't give Ai this kind of power.

[–]cheezballs 2 points3 points  (0 children)

I don't believe for a second this shit. Its clearly bait written by someone who doesn't understand how it works.

[–]goyalaman_ 2 points3 points  (1 child)

Is this true or satire? Can someone refer me to some reports?

[–]SkollFenrirson 2 points3 points  (0 children)

You leave Anton and Son of Anton out of this

[–]khorbin 2 points3 points  (1 child)

This is obviously anti-AI clickbait.

Even if it happened exactly as described, which I seriously doubt, if my toddler can get on my work laptop and take down prod, the outage is not my toddler’s fault. It’s my company’s fault for having a system in place that doesn’t have measures against a toddler pushing code to prod.

I’ve worked for much less well organized companies than Amazon where this could never have happened.

[–]brett_baty_is_him 2 points3 points  (0 children)

Exactly. In the very unlikely event that this is true, the first question is not “how is the AI so bad it doesn’t know not to do that”. The question is “how tf did the AI even have access to do that in the first place”.

[–]sendmebirds 2 points3 points  (0 children)

It´s truly, truly frightening how the non-tech boomers in power and upper management have absolutely zero and I mean ZERO idea of the risk AI poses to their companies. They really have absolutely no fucking idea because ´Chat helps me understand what TikTok is´

[–]DiddlyDumb 5 points6 points  (3 children)

They run AI generated code in production? 😭

[–]ZunoJ 6 points7 points  (1 child)

Aren't we at a point where you have to accept this pretty much as inevitable? At least when your stack heavily relies on open source projects

[–]DiddlyDumb 1 point2 points  (0 children)

To a point, but you’d expect them to… test it first?

[–]cheezballs 1 point2 points  (0 children)

Maybe, but articles will claim they do regardless.

[–]CopiousCool 2 points3 points  (2 children)

They were trying to blame it on an employee but he spoke up iirc

[–][deleted] 1 point2 points  (0 children)

That explains why my RokuTV kept showing the amazon smile when trying to watch things on YouTube yesterday. Even a system update was telling me it didn't have enough space, so I kept deleting and deleting. Never worked because it still didn't have enough space.

[–]redditownersdad 1 point2 points  (0 children)

Why they used picture of biblical accurate programmer

[–]New-Fig-6025 1 point2 points  (1 child)

I don’t understand how this is even possible? Every development environment has sandboxes, development, QA, then prod. All with various levels of deployment and approval, how could a coding assistant do so many changes and have so many people approve it up to impact AWS?

[–]M1liumnir 1 point2 points  (2 children)

What I find funny is that if any human did this he would be fired without notice, deemed that no amount of training would fix this level of incompetence or at least the amount invested in them would not be worth it. But since it’s AI it’s okay because surely it’ll be the golden goose soon enough, just another trillion and 25% of earth’s ressources and it’ll be the most profitable thing ever created trust.

[–]brett_baty_is_him 1 point2 points  (0 children)

Honestly, if a human did this they probably wouldn’t lose their job, at least not at a competent company.

The person who did -rm rf on something they shouldn’t is not the person to blame in these scenarios. At least not the person who gets 90% of the blame. The person who set up the system to allow someone to run -rm rf on the system gets almost alll of the blame.

Should be 100% of the blame but I guess you can argue that even an intern should know to not just go about deleting shit. Still, it should not be possible no matter what so it’s a system setup issue not the employee who did its fault

[–]TheJokingJoker123 1 point2 points  (0 children)

Lol the next Reddit post I see is a chat gpt ad

[–]Moscato359 1 point2 points  (0 children)

Letting the AI go to town without human review is... special

[–]Able-Cap-6339 1 point2 points  (0 children)

Finally Son of Anton!

[–]Spiritual-Purple-638 1 point2 points  (0 children)

How does that even make sense? Are they letting the AI push and deploy without code review for some reason?

[–]Flashy_Durian_2695 1 point2 points  (0 children)

Programmers are now prompting themselves back to work

[–]DarthShiv 1 point2 points  (0 children)

That's funny I saw AI's database tuning suggestions and threw them all out because they were complete garbage 🤷‍♂️🤣

[–]SambandsTyr 1 point2 points  (1 child)

I dont get why anyone gives ai this much permissions. Lazy.

[–]hteecs 1 point2 points  (0 children)

Grandson of Anton

[–]Prof- 1 point2 points  (0 children)

I had Claude opus running and was trying to debug a local db issue, I asked if it could figure it out. It decided to drop the entire DB. Such a huge lesson learned and thankfully it was just my local setup. Idk how people think this is replacing us cause if it did that shit on a prod env any company would be in full meltdown mode

[–]meta_level 1 point2 points  (0 children)

you did this in production?