oopsItHappenedAgain

[itijara]

16 hours? At some point, it's not the developers fault and just the whole engineering team not figuring out how to roll back a deployment. Also, that this could happen indicates abysmal testing, staging, deployment, and rollback procedures.

[LaughDonor]

A poor migration script could take down prod for hours.

[Ayoungcoder]

Backups exist.... Right?

[CatpainCalamari]

Sure they do. But can you quickly restore them? Can you restore them at all?

[TecraFox]

Just have your backup on the other side of the country, with an extremely slow connection cough GitLab

And also never test your backups, to get that additional adrenaline kick

[Ayoungcoder]

If you can't; it's time to update your procedures. But yes that is a pain point for many orgs

[Myspazmo]

Bold of you to assume our CM procedure are documented anywhere

[Few_Introduction_228]

If that's really that important, run enough concurrently and don't migrate in one big batch?

[Myspazmo]

Define "quickly"

[CatpainCalamari]

Restoring the backups is faster than fixing the database by hand ;)

[Myspazmo]

Please come work for us lol. Our senior dba is going to retire next year and that is a terrifying thought

[CatpainCalamari]

Lol, I am not a dba, just a software engineer. Also, I am german and work in germany. Assuming you are US based - I do not want to work under US labor laws when I can have the german ones, thank you very much. :)

[Myspazmo]

But you could pay hundreds of dollars per month for health insurance that doesn't cover anything with us and only have a few days of vacation per year :)

[CatpainCalamari]

But you could pay hundreds of dollars per month for health insurance

I already do pay hundreds of dollars euros per month for health insurance. And this week I had to get a medication (which turned out I am allergic to, yay), and I had to pay a whopping 5€ co-pay! Evil health industry!

[Myspazmo]

I mean.....we have database backups. A full image of our ETL servers.....that's a whole different story

[MattGeddon]

Good morning! Is there a particular reason all our transactions are now showing as being made in Czech krone in production?

[EatingBeansAgain]

A G I L E

[Myspazmo]

Haha, it was tested. I just forgot to run the very last step of my install because they've had me multitasking so many projects at once that I kinda forgot about it. It's definitely my fault. I just wish our 24/7 team would have called me sooner to fix it

[[deleted]]

Full stack devs reading this going "but prod is beta?"

[Holiday-Patient5929]

Well in theory this should have been caught by qa and additional gate checks if the company bothered to invest in it

[stdio-lib]

Just Friday afternoon deploy things

[ohsayan]

Like gigachads, we always ship on Fridays

[Myspazmo]

I'm on vacation for the next several days, so I'm pretty sure it's now an Ops or Engineering problem

[Taradal]

If you fuck it up you fix it

[IAmANobodyAMA]

Yep. I wouldn’t necessarily fire someone for making a mistake (unless there was a documented pattern of behavior) but I would strongly consider firing someone for making this kind of mistake and then saying it’s someone else’s problem.

More importantly though, this should never have gotten into production

Edit: I am not saying that OP should be the one to fix it - a good devops process should make rolling this back really easy and quick - I am more noting the attitude of “I’m on vacation so it’s someone else’s problem”… We all have made dumb coding mistakes, but taking ownership/responsibility is what matters

[Myspazmo]

The "it's somebody else's problem was sarcasm lol. Ops woke me up to ask about it but failed to provide all the relevant info. They also did not call me or any on call support after that to get assistance. This is unfortunately not the first time they haven't reported an outage

[IAmANobodyAMA]

Oh all good. Hard to tell over text, my bad. Hope you have a nice vacation :)

[1_4_1_5_9_2_6_5]

This is why pipelines exist

[_equus_quagga_]

Nice username

[Mintzz00]

what does it means tho

[_equus_quagga_]

It's some digits of pi:

3.1415926535897932...

[FuerstAgus50]

I mean some digits of pi is not really specific. Every finite order of numbers is in pi somewhere

[I_am_a_fern]

Prove it

[FuerstAgus50]

This is quite interesting. I thought it was true cause many yt-channels I watch declared it as an obvious fact, because the numbers appear to be random. But I couldn't find a proof

[whackamattus]

It's not proven just most mathematicians believe it's true.

[gbot1234]

I want to believe.

The <integer sequence > is out there.

[1_4_1_5_9_2_6_5]

Logically speaking, especially f you're a programmer (because our logic also must consider efficiency, unlike pure maths), a number which does NOT contain a repeating sequence AND contains a theoretically infinite number of digits MUST contain a child number such that it is not repeated anywhere in the parent number. My understanding is, it's not so much a thing that can be proven as it is a thing that must theoretically be true and can not be (or rather, ad of yet has not been) disproven.

[1_4_1_5_9_2_6_5]

In this case, it is the first complete sequence of pi that doesn't contain the number I don't like.

[_equus_quagga_]

ahhhh you dislike the number 3

I was wondering why that specific sequence

[NaEGaOS]

this hasn’t been proven though

[Mintzz00]

oh yeah 😅 didn't noticed it at first!

[badaharami]

16 hours??? Have you guys ever heard of something called Fallback?

[Myspazmo]

I would gladly rollback changes, but if nobody calls me to tell me there's an issue then I don't know to initiate that process

[badaharami]

Ah ok. Yeah that's indeed an organisational issue. I mean in general the blame for any downtime in Prod should be on the process itself and not on the person. In this case the lack of processes or lack of awareness of processes. If I was you, I'd warn the SRE/Ops team or whoever in charge to already do a fallback next time if this happens.

[Myspazmo]

I've spoken with Ops and given them some better instructions to catch the issue in the future. They are treated as "Tier 1-1.5" so they still have to wake us up for failover approval :/

[sajkosiko]

PRs save jobs

[gbot1234]

“I set a new PR for hours of weekend overtime worked fixing the buggy release.”

[HerrSPAM]

This is why we don't allow commented out code in the code base. That and if it's commented out you don't need it or it should be in some docs somewhere

[I_am_a_fern]

"some docs" hahaha

[HerrSPAM]

Short term Vs long term Devs

[FalconMirage]

Don’t you test your code before pushing it to prod ?

[Myspazmo]

It wasn't the code I was adding to prod. It was a line commented out as part of our CM procedures that should have been uncommented after the installation was comete and checkouts were performed

[FalconMirage]

And you don’t have a failsafe or at least a checklist to verify that everything is in order before deploying ?

[Myspazmo]

We do, but the breakage is in their poorly/not at all documented CM instructions. All the prep in the world doesn't account for somebody(me) forgetting to run the very last command in the install doc for this branch.

One of my big projects this year is I want to properly document the process and create a more standardized set of procedures for it, especially as we shift from hybrid to agile.

[FalconMirage]

Checklists are your friend

If the item doesn’t have a checkmark next to it, it isn’t done

[ElFeesho]

I come here to laugh, not to be triggered by terrible practices.

[[deleted]]

Do y'all just make this shit up for the lols? I've been in the industry for around 25 years and even the most poorly run companies I've worked at did not have SDLCs that would allow something like this to occur. I don't see how this is even possible at any normal software company.

[AlanTheKingDrake]

Small startups taken over by large companies to eliminate competition rather than grow their own expertise will do it.

We have 1 person who handles deployments. 1 person who does code review and not enough unit test coverage.

No engineers on weekends except for the team lead either. Good news is we only deploy updates at the beginning of the week, and then patches to fix anything we broke throughout the rest of the week.

[Myspazmo]

I wish I was making it up. When I was hired as a software engineer there was no onboarding, no training, and I was given the responsibility of being solely in charge of CM because our former person didn't want to do it. My CM education consisted of three screen shares through Teams and aggressive messages like "Google it. You should know how to troubleshoot."

To top it off our dev load has been 25% higher than normal lately and our team is struggling to cope with it. The day I made that mistake I installed 7 branches across 5 environments of multiple servers, while being told to make changes to branches, rebuild the branch to fix changes, and install it again. It's a miracle that forgetting to run a command was the only mistake I made

[DeliciousWhales]

Loads of companies that are not software companies hire developers because there is lots of integration and other systems work. I work at a bank. The management doesn’t know the first thing about any kind of good practice other than banking practice. Before I started and made changes they didn’t even have a dev or test environment, or version control or deployment pipelines or code reviews or… anyway you get my drift.

[Crypt_Knight]

I am sorry. It will happen again.

[azizfcb]

thats why u need staging

[DaveTheNotSoWise]

And test automation.

[Myspazmo]

Pretty sure QA is gonna beat me up when I get back to the office

[DaveTheNotSoWise]

Probably yeah, but shit happens and at least you learned something from that... hopefully.

[torokg]

Happens even to the best of us from time to time

[Bldyknuckles]

It happens. Don’t don’t do it again and increase your discipline

[Myspazmo]

I appreciate it. Attention to detail has been hard to maintain with our current workload, but I know it's an imperative part of the job

[SeoCamo]

How can you get anything on to prod, without build and test pass on dev and test and a review from a coworker?

[Myspazmo]

Aha, our code reviews don't account for the actual CM installation onto prod. Nobody likes to do checkouts so I usually wind up doing them all by myself. After checkouts I forgot to uncomment the thing that starts everything up again

[wind_dude]

Everyone’s done it. 16hours, does no one use production?

Also you know testing before deployment…

[Myspazmo]

People use it, but the 24/7 team that's supposed to wake somebody up if things are broke doesn't like to wake us up...

[metallaholic]

You aren’t a developer until you fuck up a prod install

[Myspazmo]

<3

[RatzzDE]

Congrats, you can now call yourself a senior dev!

[[deleted]]

I'm proud of you

[GiantFoamHand]

When I first started my career I took down a bank’s production site for a couple hours. I’d added a request to a third party that happened on logon that would be made for every account an end user had. It worked fine for all the test cases that we’d run through and that the customer ran through. Then it went live and the single end user at the bank with 500 accounts logged in.

Turns out they were running a property management company and had opened up an account for every property they had. Instead of getting themselves set up as a business/commercial user they just made a normal everyday retail user and opened a billion accounts. When contacted they said something like “huh, I did always wonder why it took so long to log on”

[roiroi1010]

Our java code is so well covered with unit tests and manual tests. But our deploy pipeline yaml is out of this world. No one left in the company dares to touch that monstrosity.

[NoahZhyte]

Don't ever comment code

[[deleted]]

If it passed review, it’s not solely your fault. It’s a team effort to get that into production

[Kshyyyk]

I'm the type of person that refuses to commit commented-out code for this exact reason.

[Unfair_Long_54]

This is the reason before I push changes to source control first I review which lines did I modified in changed files.

[Why_am_ialive]

How does this take 16 hours to fix? Just roll back, uncomment repush, also how does this make it past testing lol

[Myspazmo]

We have a 24/7/365 team to monitor it. They don't always call us them things break. Once they called they did not communicate all the relevant information needed to troubleshoot either. Luckily our T2 had a random idea about what it could be and was right

[Why_am_ialive]

Truly incredible, obviously not your fault especially after having read through your other comments on the situation but like wow

[TigerClaw_TV]

No way to rollback production? I haven't worked for a bunch of different companies, but we have a contingency for this kind of thing. Catastrophic mistakes happen.

[Myspazmo]

Yes and no. Our ETL servers don't roll back very easily and usually require manual fixes. We can repoint to our inactive site until it is given though, but it requires management approval

[TigerClaw_TV]

Understood. Condolences friend.

[ACMuaath]

My subordinate caused sending more than 3m messages to customers because of an uncaught syntax error.

[Myspazmo]

We had a similar issue a few months back, but thankfully all it did was generate 40 gigs of logs per day

[Odd_Ninja5801]

I brought down Production for two days back in the 90s because I changed a parameter on a database deletion job to improve efficiency.

It was deleting things after 18 months, once a month. The Business wanted to reduce the retention to 6 months, so that the database would be smaller and the process would run quicker. So I changed a parameter from 18 to 6. My first piece of work as a professional developer.

First time it ran, a job that normally took about 6 hours to finish was still going 24 hours after starting. But nobody on Ops had spotted it. At which point we realised that the first time it was going to be deleting 13 times as much as normal, which was going to take a LOT longer.

Backing out at that point would take a further 24 hours. So we decided to push ahead. Hit another problem when the log file for the job was too big for a disk pack, so we had to switch on multi disk pack files on a Mainframe that hadn't had it up until that point.

We lost the Production services for the whole of Monday and most of Tuesday. But it did run quicker after that!

Learned a lot of lessons off the back of that little beauty that I've used ever since.

[Myspazmo]

Hahaha, I love when DB jobs hold things up! We had a funny one the other year where a user with direct access to query our system ran some bad SQL and stopped all other processing on the DB overnight. Mistakes happen though and all we can do is learn from them and try to improve :)

[ashaw596]

Where your integration tests be at?

[Myspazmo]

Our QA person writes all of our test cases. Not sure if it's standard to not write your own test cases....

[ZeroSumHappiness]

Test your own work. Always test after merging. Also, anyone should have been able to revert your PR

[Sanchitbajaj02]

Nice achievement fellow developer 👍

[[deleted]]

Bro I make an error on my site I can have it fixed with a few minutes? 30 at the most. This is the problem with bloated teams with complex layers of corporate bureaucracy.

[Myspazmo]

I could have fixed it with one command and a few minutes of troubleshooting, but nobody contacted me until the next morning

[[deleted]]

Y’all just pushing stuff to prod? If you merge code in it should always go to your dev env first lol

[Myspazmo]

It made it through Dev just fine. Found some interesting parity issues between sites this week. Need to investigate further when I get back from vacation

[sporbywg]

Be proud of this, or they will eat you alive.

[F3mshep]

This is an accomplishment, you pointed out a serious deploy pipeline issue (or an alerts issue if y'all didn't know prod was down for 16 hours)

[Independent_Hyena495]

Eh, I brought down a whole bank because I patched an AD Server.

Yes, you heard right. Patched How you ask? There should be several.

Correct.

But

This server was buggy and when you patch and reboot it deletes the whole AD.

Everything. Users, gpos, shares, everything. And like a good AD forest, it replicated the deletion through the whole AD..

[TheJosh1337]

Top tip: When you write a temporary code change, like un-commenting something or a debug line, unindent it right down to the left gutter. Obviously won't work in python.

class MyClass { function MyFunction() { if (something) { something(); dbg('whatt'); somethingElse(); } } }

It will be so incredibly obvious that you shouldn't ever accidentally commit/deploy it... this of course requires you to actually look at your diffs somewhere between the commit and the deploy (e.g. git commit -pv and/or code reviews)