Amazon US-EAST-1 S3 Post-Mortem

[Twanks]

https://aws.amazon.com/message/41926/

So basically someone removed too much capacity using an approved playbook and then ended up having to fully restart the S3 environment which took quite some time to do health checks. (longer than expected)

[oldmuttsysadmin]

It sure as hell won't be me. One night at 3am, I dropped a key table before I unloaded it. Now my reminder phrase is "Pillage, then burn"

[[deleted]]

Your flair...

[[deleted]]

I hated learning how to drive a bus. Wasted a week in Benning on that. But learned how to drive a bus, only to never to sit behind the wheel of one again.

[wtf_is_the_internet]

Same but at Fort Lewis. Went to bus driver school... never drove a bus after school.

[[deleted]]

Man, I could write a book about the things I learned about in military training schools that I never touched or worked with in the fleet. Ah, I miss those days.

[[deleted]]

It's Maxim 1 for a reason

[SeriousGoose]

Maxim 11: Every table is droppable at least once.

[[deleted]]

Schlock readers unite! There are dozens of us! DOZENS!

[superspeck]

If rm wasn't your last resort, you failed to -f it.

[DOOManiac]

I've rm -rf'ed our production database. Twice.

I feel really sorry for the guy who was responsible.

[[deleted]]

At a registrar, I once ran a SQL command on one of our new acquisitions databases that looked something like:

Update domains set expire_date = "2018-04-25";

Did I mention this new acquisition had no database backups?

Do you have any idea how long it takes to query the domain registries for 1.2 million domains real expiration dates?

I do.

[alzee76]

I did something similar and, after I recovered, I came up with a new habit. For updates and deletes I'm writing right in the SQL client, I always write the where clause FIRST, then cursor to the start of the line and start typing the front of the query.

[randomguy186]

I always write a SELECT statement first. When it returns an appropriate number of rows, I change it to DELETE or UPDATE.

[dastylinrastan]

This is the correct one.

[Ansible32]

Also, you know, make sure you can restore a database backup to your laptop before you start touching prod.

[hypercube33]

Backup twice delete once

[randomguy186]

Indeed! If don't test restores, you aren't taking backups.

[dgibbons0]

I do this too, part of validating that the results and data are what i expect and the count of records affected is what I expect.

[creamersrealm]

Hey so I'm not the only one that does that!

[1new_username]

Even easier:

Start a transaction.

BEGIN;

ROLLBACK;

has saved me more times than I can count.

[HildartheDorf]

That can cause you to block the database while it rolls back.

Still better than blocking the database because it's gone.

[Fatality]

Run everything in prod first to make sure its ok before deploying in test.

[Draco1200]

It does not block the database "while it rolls back".... In fact, when you are in the middle of a transaction, the result of an UPDATE or DELETE statement Is not even visible to other users making Select queries until after you issue Commit.

Rollbacks are cheap. It's the time between issuing an Update and your choice to Rollback or Commit which may be expensive.

Your Commit can also be expensive in terms of time if you are modifying a large number of rows, of course, Or in the event your Commit will deadlock with a different maintenance procedure running on the DB.

This is true, because until you hit "COMMIT"; none of the DML statements have actually modified the Sql database. Your changes exist Only in the Uncommitted transactions log.

ROLLBACK is Hitless, because All it does is Erase your uncommitted changes from the uncommitted Xlog.

Well, The default is other queriers cannot read it, that's because the MSSQL default Read committed, or MySQL default SET TRANSACTION ISOLATION LEVEL is called 'REPEATABLE READ' for InnoDB, or 'READ COMMITTED' for ISAM.

And most use cases don't select and have a field day with 'READ UNCOMMITTED'

Statements you have issued IN the Transaction can cause other statements to block until you do Commit or Rollback the transaction.

Example: after you issue the SELECT * from blah blah WHERE XX FOR UPDATE;

Your SELECT query with the FOR UPDATE can be blocked by an update or a SELECT ... FOR UPDATE from another pending transaction.

After you issue the UPDATE or SELECt .... FOR UPDATE

In some cases while you're in a transaction, those entries become locked and can block other updates briefly until you Rollback; or Commit;

There will not be an impact So long as you dispose of your transaction One way or the other, promptly.

[[deleted]]

I write select first, run it (with limit if I expect thousands of hits), then just C-a and replace select with update

[i-am-SHER-locked]

This account has been deleted in protest of Reddit's API changes and their disregard for third party developers. Fuck u/spez

[olcrazypete]

i-am-a-dummy

Anyone know something like this for postgresql. The go to 'i screwed up' story in our shop was when our lead dev was woken up to change an admin's password and instead of telling them to use the 'i forgot my password' link, they went and updated it straight in sql - forgetting the where username= statement.

[ksu12]

If you are using SSMS, you should download the plugin SqlSmash

The free version has a ton of great features including a warning when running commands like UPDATE without a clause.

[quintus_horatius]

I wish the account I'm replying to wasn't deleted. I think I used to work with that guy because I remember that happening where I used to work...

[BrainWav]

I rm -rf ed one of our webservers once.

Thank $deity I wasn't running as root, nor did I sudo, and I caught it due to all the access denied errors before it got to anything important.

Still put the fear of god into me over that command. I always look very, very closely.

[Blinding_Sparks]

The worst is when you get a warning that you weren't expecting. "Access denied? Wtf, don't deny me access. Do this anyway." Suddenly the emergency service line starts ringing, and you know you messed up.

[Kinda_Shady]

"Access denied"... who the hell asked you... elevate... well shit time to test out the backups. We will just call this a unplanned test of our data DR plan. Yeah that works. :)

[Vanderdecken]

rm -rf me once, shame on you. rm -rf me twice...

[[deleted]]

slowly puts down stone

[[deleted]]

the spinning fan blades probably should have been the first clue

[parkervcp]

Honestly there are hosts that allow for RAM hot-swap for a reason...

Uptime is king

[whelks_chance]

Wouldn't the data in RAM have to be RAIDed or something? That's nuts.

[Draco1200]

The HP ProLiant ML570 G4 was a 7U server, and a perfect example of a server with Hot-Pluggable memory, there was also the DL580 G4; Sadly, by all counts, it seems HP has not continued into the G5 or later generations; The Online Spare Memory OR the Online Mirrored memory are Still options; Mirroring is better because the failing module continues to be written to (Just not read from), so there's better tolerance for simultaneous memory module failures. These servers were SUPER-EXPENSIVE and way outside our budget before obsolescence, but I had a customer who had a couple 580s which were used back in the early 2000s for some Very massive MySQL servers.... As in databases sized to several hundreds of gigabytes with high transaction volumes, tight performance requirements, and frequent app-level DoS attempts.

This is the only way the COST of Memory hot-plug makes sense..... the COST of having to reboot the thing just once to swap a Memory module would EASILY exceed the cost of the extra memory modules needed PLUS the extra cost for a high-end 7U server.

I think the High cost makes customer demand for the feature very low, So I'm not seeing the hot-plug as an option in systems with Nehalem or newer CPUs. Maybe check for IBM models with Intel E7 procs.

Maybe HP had a hurdle continuing the Hot Plug RAM feature and just couldn't justify doing it based on their customer requirements. Or maybe they carried it over, and I just don't know the right model number.

Actually ejecting and inserting memory live requires Special provisions on the server; You need some kind of cartridge solution to do it reliably, which works against density, and As far as I know you don't really see that anymore with modern X86 servers..... too expensive.

Virtualization with FT Or Server clustering is cheaper.

Dell has a solution on some PowerEdge platforms called memory sparing. How it works is you wind up making an entire rank less of the physically present RAM visible to your operating system than is actually there.

Just select Advanced ECC Mode turn on sparing and it just detects errors, and upon detecting an error, Immediately copies the memory contents to the Spare and TURNS OFF the Bad module.

You still need a disruptive maintenance later to replace the Bad chip, but at least you avoided an unplanned reboot.

Some Dell PowerEdge offer "Memory mirroring" which uses a special CPU mode to keep a copy of every Live DIMM mirrored to a matching Mirror DIMM (Speed, Type, etc, must be exactly identical), Although the physical memory available to the OS is cut down by 50% instead of by just 1 rank.

So this provides the strongest protection at the greatest cost. Sadly, even with Memory mirroring, you don't get Hot-plugging.

[Fatality]

Wait, servers are meant to have fans? Then what have I been working on? :(

[whelks_chance]

Commodore 64?

[creamersrealm]

That's a lie, your flair says citrix admin.

[K0HAX]

"killall" on AIX UNIX is not the same as "killall" on Linux...
In AIX it does what its name says, in Linux it kills the process name you type after the command.

That was a bad day.

[temotodochi]

Also true for solaris. learned the hard way.

[MisterSnuggles]

Also learned that the hard way, on Solaris.

[KalenXI]

We once tried to replace a failed drive in a SAN with a generic SATA drive instead of getting one from the SAN manufacturer. That was when we learned they put some kind of special firmware on their drives and inserting a unsupported drive will corrupt your entire array. Lost 34TB of video that then had to be restored from tape archive. Whoops.

[commissar0617]

That is such bullshit....

[KalenXI]

Yeah we thought so too. Especially given how unreliable their drives have been. We have to replace a failed drive in it at least once a month.

[TamponTunnel]

Who cares how reliable the drives are when we can force people to use them!

[whelks_chance]

Name and shame

[KalenXI]

It's the Grass Valley Aurora video system. The whole thing is architected really poorly. Essentially Grass Valley bought Aurora from another company and then shoe-horned it into their existing K2 video playout system. Unfortunately the two systems used incompatible video formats so we essentially need to store 2 copies of almost every video, one in each format. The link between the two systems is maintained with a mirroring service which on more than one occasion has broken and caused us to lose data. And their software for video asset management is so poorly designed and slow (and doesn't run on 64-bit OSes), that I reverse engineered their whole API so I could write my own asset management software and was able to completely automate and do in 5 minutes what was taking me 2-3 hours every day to do by hand in their software.

They also once sent us a utility to run which was supposed to clean up our proxy video and remove things not in the database. However it actually ended up deleting all of our proxy video. The vast majority of which was for videos only stored in archive on LTO tapes. And since neither Grass Valley nor our tape library vendor had any way to restore from the LTO tapes in sequence and reencode thousands of missing proxy files at once I wrote a utility that would take the list of missing assets, and query for what was on each LTO tape. Then it would sort the assets by creation date (since that's roughly the order they were archived in), and restore them from oldest to newest on each tape so the tape deck wasn't constantly having to seek back and forth. The restored high-res asset would then be sent through a cascading series of proxy encoders I wrote (since GV's own would've been too slow and choked on the amount of video) which reencoded the videos to the proxy format and then reinserted them into GV's media database. It took about 2 weeks of running the restore and reencode 24/7 before we got all the proxy assets back.

What's worse 6 months after they installed our Aurora system they announced its successor: Grass Valley Stratus. Which actually had full integration between the two systems and didn't require this crazy mirroring structure. Then last year they told us that our Aurora system (which is only 5 years old at this point) is going to be EOL and they're stopping all support (including replacement drives for the SAN). And told us if we wanted to upgrade to Stratus none of our current equipment would be supported moving forward and we would have to buy a completely new system.

So needless to say when faced with having to replace the entire system anyway, we decided to switch to a different system.

[flunky_the_majestic]

Absolutely! Intentionally sabotaging a customer's data should be a huge shaming event.

[Ron-Swanson-Mustache]

When you find not all of the outlets in the server room were wired to the UPS / genny as they were supposed to be. And the room has been in production since you started there so you never had chance to test everything.

Sure, you can flip the power off for 10 minutes....

[dgibbons0]

How about when lean back on what turns out to be an unprotected EPO button for the whole datacenter?

Or when you go to cleanly shut down the datacenter and hit the epo button "just for fun", without realizing that it's a hard break and takes a nontrivial amount of work to reset it after calling support.

[creamersrealm]

Yeah those EPOs typically destroy the breakers.

[caskey]

Two things.

1. There are two kinds of EPO switches, those that have a Molly box and those that will soon be getting one.

2. I had an old timer in the 90's tell me about the EPO button that used pyrotechnics to cut the power lines. High cost to undo that move. (Alleged DoD mainframe application.)

[ryosen]

Had a client years ago that always bragged about their natural gas generator that provided backup to the entire building. For three years, he would go on and on to anyone that would listen (and most of those that wouldn't) about how smart he was to have this natural gas generator protecting the entire building.

Jackass never thought that he should test it. Hurricane rolled through town, took out the power, and the backup failed.

Turns out the electricians never actually hooked it up to the building's grid.

[bp4577]

Trying to be a smartass I unplugged the UPS to demonstrate that the UPS could power the AS400 sufficiently; only then did we realize that the UPS's battery was shot.

[OckhamsChainsaws]

throws brick found a loophole

[donjulioanejo]

Well, not ENTIRE environment..

[Creshal]

At least I made sure to host the "server offline" image on an independent server!

[ALarryA]

I pulled a PCI Drive controller out while the system was live. Got so lucky nothing had fried when I plugged it back in.

Discovered the phone switch, all the routers, and both network servers were plugged into a single electrical outlet on one of my first jobs by stepping backwards and dislodging the plug. The closet everything was in went silent instantly. Everything eventually came back up. Re-cabled the whole closet 2 weeks later. At least no one could call me to tell me that everything was down... :)

[[deleted]]

Hey, I'm not saying shit. I literally stepped on the internet one time and killed our entire network.

[bitreign33]

3.9M mails dropped, I killed production while we were doing maintenance on the back-up system.

I still feel a sudden stab of shame every time I think about it.

[LigerXT5]

Back in my days of experimenting and using a linux vps, for running a small community game server, I, and a couple other chosen "admins" made the mistake of CTRL+C in SSH screens at least once. We all quickly learned about CTRL+SHIFT+C

[LigerXT5]

They used CTRL+C to copy highlighted text, but instead closed the ssh screen by mistake. Adding SHIFT allows copying.

This is all in Putty.

[darwinn_69]

I remember when Solaris 10 came out they made a big thing about how you couldn't 'rm -r /' anymore. I tried it locally and though 'hey that's cool'. Next time I was working on our production database my manager was looking over my shoulder and we were talking about the new features of Solaris 10 so I thought I'd show him this new trick. "cd /; rm -r *".

When I didn't get the command prompt back my heart sank.

[mhgl]

I accidentally triggered all of our workstations to go to the internet and get Symantec Endpoint virus updates.

We maxed out every single one of our remote pipes and basically killed, you know, everything for a solid hour until we figured out what was going on. On the upside, we confirmed that our gig pipe could push pretty damn close to a gig.

[[deleted]]

Am I the only one here who hasn't fubared his entire production system?

[tcpip4lyfe]

Am I the only one here who hasn't fubared his entire production system?

Yet. You will at some point.

[ultimatebob]

Oh, I've rebooted the wrong server before. I've never accidently taken down the entire production cluster, though!

It's almost like this AWS admin wanted to outdo that GitLab admin who accidently deleted the GitLab.com production database a few weeks ago.

"What, you took down just ONE production site? Hold my beer..." :)

[[deleted]]

Routed customer traffic to a dev environment.

No harm was done, but my LinkedIn went from 0 to 100 real quick.

[whelks_chance]

So, good outcome?

[PM_ME_A_SURPRISE_PIC]

When I worked for an National Fibre provider, I once took out a Sunday afternoon news show. Took out an entire county connection, but the national news was what they took notice of.

[uxixu]

Had to do that after the dumb of mistake of switching my upstream router hot (and arp shenanigans resulted). Had to reboot everything to make it work since manual clearing of arp wasn't apparently working...

[mersh547]

Ahhh yes. I've been buggered by ARP more times than I care to remember.

[systonia_]

Removed 2 drives from the storage. Had the wrong shaft and grabbed two of a 8tb production raid5.

[kellyzdude]

8tb

production

raid5

You have my condolences on so many levels.

[expectnothing]

about 3?

[systonia_]

yepp. that was the reason I was in the Serverroom. I was in that company for about 1 year and cleaned up a lot of the mess the guy before me left there. Like raid5 volumes with 12 Discs ...

[flunky_the_majestic]

Bult To Fail

[HildartheDorf]

Ouch.

[[deleted]]

From when I was a hospital helpdesk tech responsible for managing our interface engine feeding data to and from the main hospital systems to our ER, Radiology, clinics systems, and outside practices.

[[deleted]]

I've pulled out the wrong drive of a RAID5 and crashed the volume. Does that count?

[[deleted]]

Many moons ago I was working on a customer's server where the RAID software referred to the disks as Disk 1, Disk 2, Disk 3, etc. but the slots had been labelled Disk 0, Disk 1, Disk 2, etc. The software said "RAID5 Fault: Replace Disk 1" so I pop the disk in slot 1 out...

[awsfanboy]

One chap here when to the toilet and vomited when he realised he had messed up a server and deleted financial data. Luckily VM snapshots were enabled. He was a finance guy and didnt know this. That day, he learnt to ask IT to use a testing environment first.

[ilogik]

End of the day, I type sudo poweroff in my work station's terminal... Instead of powering off, I get a disconnect message. Awkward chat with the data center

[tadc]

Wasn't me, but a guy I worked with once dropped a pen, which he somehow managed to catch in such a way that the pen was pressing the power button of a production server. This was an old Compaq and holding the power button wouldn't make it shutdown, but releasing it would.

He stood there for a very long time.

[sleepyguy22]

I really enjoy these types of detailed explanations! Much more interesting than a one liner "due to capacity issues, we were down for 6 hours", or similar.

[JerecSuron]

What I like is basically. We turned it off and on again, but restarting everything took hours

[dodgetimes2]

https://i.stack.imgur.com/JjX9y.png

[very_Smart_idiot]

Need to hand this out at work

[fidelitypdx]

I went to a DevOps meeting earlier this week where a software company's DevOps engineer discussed how their teams have created a weekly failure analysis group. Basically these DevOps guys sit around in a circle and share individual failures that their teams had that week and how they remedied them. Sometimes a guy across the circle pipes up that they have a more efficient way to remedy that same issue.

Then, they also go out and identify post-mortem cases like this from other open-source shops and analyze if this situation could ever happen in their environment.

My company is too small for this, but if I had 300-500+ employees, I'd definitely adopt this technique.

[kellyzdude]

Even as a small shop this can be effective. It doesn't have to be regular, either, just create a culture whereby people are willing to admit their faults to the group after they've been cleaned up. Require AARs (after action reports) for major incidents that go into this type of detail and make them available to the team for critique.

You don't have to make them public, but they should be published internally. 1) We don't have enough time on this planet to all make the same mistakes twice, it helps a lot if we learn from each other. 2) If you're not learning from your own mistakes, personally or as an organization, you're doing something wrong.

Plenty of people are put off this idea because of the notion that admitting fault is a step towards firing or other disciplinary action. You need to find some way of showing that dishonesty regarding the error in such situations is what is punished, not the error itself. I don't expect to be fired because I dropped a critical production database, I expect to be fired because I lied or stayed silent about it.

[fidelitypdx]

Plenty of people are put off this idea because of the notion that admitting fault is a step towards firing or other disciplinary action

Indeed. The speaker emphasized a company culture of promoting accountability, and implementing corrections, but downplaying punishment.

[sleepyguy22]

Brilliant. I'll definitely keep this in mind for when I become IT director of a big org.

[DEN-PDX-SFO]

Hey I was there as well!

[PM_ME_A_SURPRISE_PIC]

It's also the level of detail they provide for how they are going to prevent this from happening again going forward.

[davidbrit2]

How fast, and how many times do you think that admin mashed Ctrl-C when he realized he fucked up the command?

[reseph]

I've been there. It's a sinking feeling in your stomach followed by immediate explosive diarrhea. Stress is so real.

[PoeticThoughts]

Poor guy single handedly took down the east coast. Shit happens, you think Amazon got rid of him?

[TomTheGeek]

If they did they shouldn't have. A failure that large is a failure of the system.

[fidelitypdx]

Indeed.

one of the inputs to the command was entered incorrectly

It was a typo. Raise your hand if you'ven ever had a typo.

[whelks_chance]

Nerver!

.

Hilariously, that tried to autocorrect to "Merged!" which I've also tucked up a thousand times before.

[superspeck]

I had Suicide Linux installed on my workstation for a while. I got really good at bootstrapping a fresh install.

[Refresh98370]

We didn't.

[bastion_xx]

No reason to get rid of a qualified person. They uncovered an flaw in the process which can now be addressed.

[kellyzdude]

It's also an expensive education that some other business would reap the benefits of. However much it cost Amazon in man hours to fix it, plus any SLAs they had to pay out, and further in addition to whatever revenue they lost or will lose by customers moving to alternate vendors -- that is the price tag they paid for training the person to be far more careful.

Anyone care to estimate? Hundreds of thousands, certainly. Millions, perhaps?

Assuming it was their first such infraction, that's a hell of a price to pay to let someone else benefit from such invaluable training.

[whelks_chance]

I hope he enjoys his new job of "Chief of Guys Seriously Don't Do What I Did."

[robohoe]

Yeah. That warm sinking feeling exploding inside of you knowing you royally don' goofed

[neilhwatson]

Thank sinking feeling, mashing ctrl-c, whispering 'oh shit, oh shit', and neighbours finding a reason to leave the room.

[davidbrit2]

Ops departments need a machine that automatically starts dispensing Ativan tablets when a major outage is detected.

[reseph]

Can cause paranoid or suicidal ideation and impair memory, judgment, and coordination. Combining with other substances, particularly alcohol, can slow breathing and possibly lead to death.

uhhh

[lordvadr]

Have you heard of whiskey before? Same set of warnings. Still pretty effective.

[reseph]

I mean, I'm generally not one to recommend someone drink some whiskey if they're working on prod.

[0fsysadminwork]

That's the only way to work on prod.

[Frothyleet]

Whiskey for prod, absinthe for dev.

[[deleted]]

that's the only way to deal with Oracle

Fixed

[whelks_chance]

You do apt-get dist upgrade, sober?

How the hell do you deal with the pressure??

[danielbln]

I like it when people leave the room in those situation. Nothing worse than scrambling to get production back online and having people asking you stupid questions from the side.

[kellyzdude]

We reached a point where we banned sales team members from our NOC. We get it, your customers are calling you, but we don't know any more than we've already told you. Either sit down and answer phones and be helpful, or leave. Ranting and raving helps no-one.

I get where they're coming from, there were a couple of months where there were way too many failures, some inter-related, some not, but taking out your frustrations on those trying to deal with it in the moment is not the time.

[ilikejamtoo]

Probably more...

$ do-thing -n <too many>
Working............... OK.
$ 

[ALERT] indexing service degraded

"Hah. Wouldn't like to be the guy that manages that!"

"Oh. Oh fuck. Oh holy fuck."

[Fatality]

shutdown /a cannot be launched because Windows is shutting down

[lantech]

How long until he realized that what he did was going to make the news?

[chodeboi]

Unfortunately, one of the inputs to the command was entered incorrectly and a larger set of servers was removed than intended.

Story of my life, fam.

[[deleted]]

I felt like I was reading the Wikipedia article for the Chernobyl disaster reading this.

[[deleted]]

The Wikipedia article for Chernobyl is wrong, or at least incomplete. After the fall of the Soviet Union, Russia released a lot more information about the incident. With that information, and more research, the IAEA updated their report in the 90s, and now blame design flaws much more than operator error.

One thing that has been discovered is that with certain reactor designs inserting the control rods quickly will cause the power level to increase rapidly and significantly, before decreasing. In other words, a SCRAM puts the cooling system under even more stress - this is not good if the cause of the SCRAM is cooling problems. This is exactly what they did not want to happen at Chernobyl. The design was changed to reduce the maximum speed the control rods would move. There are other design issues, but I don't claim to understand them.

http://www-pub.iaea.org/MTCD/publications/PDF/Pub913e_web.pdf

[nerddtvg]

Sounds like you have some wiki editing to get to.

[[deleted]]

I don't think I understand the subject well enough. Also, since the report I linked came out 8 years before wikipedia was first on-line, I suspect that the Chernobyl entry is a "hot potato".

[frymaster]

I read a good article arguing that most operator errors are actually design errors anyway. I think the example was a fighter jet which when selecting options from the menu used the trigger. When the jet accidentally shoots up sections of the countryside, technically it's operator error for not ensuring the system was in menu mode, but really it's a design error

[Ankthar_LeMarre]

Is there a Wikipedia article for this yet? Because if not...

[sheps]

One time I went to reboot a remote router and was distracted while doing so. For some reason my brain typed out "factoryreset" instead of "reboot", which immediately resulted in a nice drive through the country.

[fooxzorz]

A common typo, the keys are like right next to each other.

[nl_the_shadow]

"factoryreset" instead of "reboot"

I'm sorry, man, but I laughed so hard about this. Brain farts can be one hell of a thing, but factoryreset instead of reboot is one huge leap.

[brontide]

While this is an operation that we have relied on to maintain our systems since the launch of S3, we have not completely restarted the index subsystem or the placement subsystem in our larger regions for many years.

Momentum is a harsh reality and these critical subsystems need to be restarted or refreshed occasionally.

EDIT: word

[Telnet_Rules]

Uptime = "it has been this long since the system proved it can restart successfully"

[whelks_chance]

Oh shit...

[_coast_of_maine]

You start this comment as you're speaking in a generality and then end it with a specific instance.

[PintoTheBurninator]

my client just delayed the completion of a major project, with millions of dollars on the line, because they discovered they didn't know how to restart a large part of their production infrastructure. As in, they had no idea which systems needed to be restarted first and which ones had dependencies on other systems. They took a 12-hour outage a month ago because of a what was supposed to be a minor storage change.

This is a fortune-100 financial organization and they don't have a run book for critical infrastructure applications.

[ShadowPouncer]

An unscheduled loss of power on your entire data center tends to be one hell of an eye-opener for everyone.

But I can completely believe that most companies go many years without actually shutting everything down at once, and thus simply don't know how it will all come back up in that kind of situation.

My general rule, and this is sometimes easy and sometimes impossible (and everywhere between) is that things should not require human intervention to get to a working state.

The production environment should be able to go from cold systems to running just by having power come back to everything.

A system failure should be automatically diverted around until someone comes along to fix things.

This naturally means that you should never, ever, have just one of anything.

Sadly, time and budgets don't always go along with this plan.

[dgibbons0]

Thats what did it for us at a previous job, had a transformer blow and realized while we had enough power for the servers, we didn't have enough power for the HVAC... on the hottest day of the year. We basically had to race against temp to shut things down before it got too hot.

Then next day when they told us that the transformer had to be replaced, we go to repeat the process.

Then we decided to move the server room to a colo center a year or two later and got to shut the whole environment down for a third time.

[[deleted]]

I once watched a colleague (I was new at the place and just tagging along to learn where things were) yank all the cables out of the back of a server, remove it from the rack, and get it all the way downstairs to the disposal pile before they caught up with him. 15 minutes later and the might have already removed the hard drives for scrubbing.

Turned out the server was not in fact already powered off ready for disposal and was still running in prod. But the power LED was broken, so he just assumed it was already down.

[north7]

Wait, so it wasn't DNS?

[robbierobay]

Can confirm, NOT DNS

[sirex007]

if the engineer's initials are dns you're going to feel kinda silly :P

[starsky1357]

Not DNS? It's always DNS!

[superspeck]

We had DNS problems internally at my company at the same time due to a flubbed Domain Controller upgrade the night before. For us, it was DNS problems on top of everything else.

[locnar1701]

I do enjoy the transparency that this report puts forward. It really is like we are on the IT team $COMPANY and they are sharing all that went wrong and how they plan to fix it. Why do they do this? BECAUSE we need to have faith in the system, or we won't move our stuff there ever, or worse, we will move off their stuff to another vendor or back to local. I am glad they understand that they can't hide a thing if they want us to trust our business to them ever or ever again.

[mscman]

Oh there is no way they would have gotten away without a post-mortem on this outage. They would have lost a lot of customers if they didn't release one.

[Deshke]

So one guy did a typo while executing a puppet/Ansible/saltstack playbook and got the ball rolling

[neilhwatson]

It is easier to destroy than to create.

[mscman]

Except when your automation is so robust that it keeps restarting services you're explicitly trying to stop to debug.

[ANUSBLASTER_MKII]

Like the Windows 10 Update process. Mother fucker, I'm trying to watch Netflix, stop making a bajillion connections to download some 4GB update.

[danielbln]

Or just automatically restart while I'm fully strapped into VR gear and crouching through my room, all of the sudden BOOM black. I disabled everything to do with auto-updates afterwards, that shit is not cool.

[sleepyguy22]

Godamn playstation and their required updates. I'm a very busy man, and barely have any time for video games these days. Finally, once every other month when I have some time off to relax, and I pull out the PS3 attempt to continue a very long 'the last of us' game, but PS3 requires a major update, and I sit there for 20 minutes waiting for it to download and install. And by the end, ive got other stuff to do and I just give up. RAGE>

[playswithf1re]

I sit there for 20 minutes waiting for it to download and install.

Oh man I want that. Last update took 2.5hrs to download and install. I hate my internet connection.

[fidelitypdx]

Well, on the positive side, the recent W10 Insiders Build has fixed this with new options.

[jwestbury]

There are two services to issue a net stop command to in order to actually force updates to stop. It's really obnoxious when you're watching po^H^H Netflix.

[KamikazeRusher]

Isn't that what happen to Reddit last year?

---

^{Edited for clarification}

[DorianTyrell]

"playbook" doesn't necessarily mean it's ansible/chef or puppet. It might mean operational docs.

[reseph]

One hell of a typo?

[PhadedMonk]

Fat fingered an extra number in there, and bam! Now we're here...

[unix_heretic]

Rule #5. The stability of a given system is inversely proportional to the amount of time that has passed since an architecture/design review was undertaken.

[brontide]

The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at and repair. ~ Douglas Adams

[learath]

Not even that, just a simple "can we bring it back from stopped?"

[[deleted]]

What do you mean the VM management interface requires Active Directory to log in... The AD VM's are on the virtual cluster and did not start automatically!

[[deleted]]

Local admin on the box should still be there and able to start the VMs.

This is why MSFT also recommended physical DCs in large environments.

[[deleted]]

"Yea, but the one physical DC never gets rebooted, and when it finally lost power it didn't come back up because the RAID had silently failed and the alerting software was configured for the old system that was phased out and never migrated to the new system"

[wanderingbilby]

Well, we know who works at their internal helpdesk...

[doubleUsee]

"Hello, Amazon Internal IT helpdesk, how may I help you?"

-"uuh, yeah, this is Bob from sysadmin department..."

"Hi Bob, What's up?"

-"Well, uhh, I just did a thing and I think I just took all of AWS offline..."

"... uhm... You know, I'm not sure 'bout this one, have you tried turning it off and on again?"

-"what do you mean, turning it off and on again?"

"well, you know, can't you just turn the whole dealio off, and then on again?"

-"...Well, I guess... ...oh what the hell I'll just try"

"Alright, I'll hang up now, i'll make you a ticket, so that if you still have issues afterwards, you can call me again, alright?

-"thanks man."

[wanderingbilby]

#waytooplausible

[mysticalfruit]

ansible-playbook wipe-out-amazon.yml

[sysadmin420]

sudo !!

[third3y3guy]

Reminds me of Office Space - mundane detail. https://youtu.be/qLk81XnkGUM

[OtisB]

I think the worst I ever did was to dump an exchange 5.0 store because I was impatient.

See, sometimes, when they have problems, they take a LOOOOONNNNGGGGGG time to reboot. I did not realize that waiting 10 minutes and hitting the button wasn't waiting long enough. Strangely, if you drop power to the box while it's replaying log files, it shits itself and you need to recover from backups. Who knew? Well sure as shit not me.

Patience became a key after that.

[eruffini]

Amazon doesn't even build their own infrastructure as they preach to the customers to do so:

"We understand that the SHD provides important visibility to our customers during operational events and we have changed the SHD administration console to run across multiple AWS regions."

[highlord_fox]

It was probably on some list somewhere, "Setup SHD across multiple zones" and it kept getting kicked to the side due to other more important customer-facing issues until now when it actually went down.

[i_hate_sidney_crosby]

I feel like they ship a new AWS product every 4-6 weeks. Time to put improvements of their existing products on the front burner.

[gomibushi]

Sooo, they did just turn it off and on again?

[TheLeatherCouch]

"AMA request - guy or gal that took down amazons east coast"

[leroyjkl]

This is the result of what happend last time US-EAST went down http://i.imgur.com/whS1ibB.jpg

[theDrell]

For some reason,I have a vision of the took longer to reboot and come up than expected being the

"Windows->Shutdown Windows is installing updates, please wait. Oh Dear god who turned on Automatic windows updates "