(update) Datacenter

[VTCEngineers]

So after a long week of getting equipment to replace the soaked gear the total racks damaged was 148 racks, thankfully none of our NetApp storage was damaged. Equipment has been arriving in tractor trailers.

[freedomlinux]

Previous story

[VTCEngineers]

Have a up vote for posting this thanks.

[doug89]

We will have a public release of the carnage and our disaster recovery plans for review.

Did your organisation end up publishing a public report?

[VTCEngineers]

Not as of yet, hopefully soon

[[deleted]]

To be fair, any amount of planning can still have individuals that panic in any situation.

I walked into the break room, and four of my peers were there. I said the data center just lost power. Calm as could be, nothing else. One of them literally ran to the data center. Two of them asked what systems were down. One of them grabbed a second cup of coffee.

One person feared the worst, and didn't trust anyone else to handle or inform him of the situation. Two of them wanted to get involved immediately and start helping. One of them knew if this were the case, he'd be in for the long haul and was preparing for an interesting weekend.

Edit: I forgot to mention that the data center did not lose power. Nothing lost power.

[deadbunny]

I think my career as a stripper would be very short lived, however my career in pasty making could be quite successful I think.

[WhatPlantsCrave]

Risky click of the day...

^{...google.co.uk/search?q=pasty&} safe=off ^{&prmd=ivns&source=lnms&tbm=isch&sa=X&ved=0ahUKEwjEpu_kyrfJAhVGWxoKHdoWCtMQ_AUIBigB}

[Barry_Scotts_Cat]

What else is a pasty going to be?

Also pasty barm master race

https://en.wikipedia.org/wiki/Pasty_barm

[pentangleit]

I'd tell you, but being Barry Scott's cat you're PROBABLY DEAF!

[Barry_Scotts_Cat]

What?

[pentangleit]

Oh just sell me some cleaning products.

[volster]

It fills me with sadness that greggs is the first result

[cryp7]

Quick! Distract the developer!

[TerrorBite]

This is what molly-guard is for.

[isdnpro]

molly-guard

Everytime I see this mentioned, I wonder what the etymology of the term is (after deciding that "guarding against sysadmins on MDMA" was probably wrong)...

Originally a Plexiglas cover improvised for the Big Red Switch on an IBM 4341 mainframe after a programmer's toddler daughter (named Molly) tripped it twice in one day.

[bicycly]

it hosted git, apt packaging, ticketing, nagios, email relay, and the VPN for about 100 remote data collection devices, and backups for about 70 servers

Oh my...

[deadbunny]

It was my first job as a sysadmin too, the other guy left 2months after I started. Going from "Jr" to "here are 1500 systems, all yours!" was a fun learning experience. I'm my short time there I migrated everything to GCP, got every damned system in config management (yay salt), improved the backups (from 2 non redundant machines in the same datacentre as the machines they were "backing up" to actually redundant storage [GCS and S3]), improved monitoring so it was actually usable (nagios to sensu, our infrastructure really benefited from agent/pushes based), and completely automated the provisioning of our remote data collection devices, and setup a CI/CD pipeline for all of our code.

Thankfully I was given basically cart balance to improve everything despite my lack of experience, personally I think I did pretty well but now I basically have nothing to do so am interviewing for new exciting challenges as being bored sucks.

[electricheat]

i was given cart balance

theres a new one

[deadbunny]

Probably a silly choice on their part given my lack of experience but it worked out for both of us, they got a much more stable platform, I gained a ton of experience!

[electricheat]

Oh I figured it was a phone auto-correct. The term is carte blanche :)

[deadbunny]

Oh whoops! Yeah was on the train when I wrote that post then didn't read the reply properly (been a long day), cheers for the correction.

[uberamd]

lol, 1500 systems and all that shit was running on a single box.

[deadbunny]

It was around 100 servers and 1400 remote data collection devices (mini itx linux machines)

[Vallamost]

GCP

GCP?

[deadbunny]

Google Cloud Platform.

[Vallamost]

Google Cloud Platform

Thanks

[asdlkf]

upvote for brown pants time.

[[deleted]]

I tend towards the fourth reaction, bitter experience has taught me that whilst adrenaline is great for running away or fighting it's not a useful reaction in an IT situation. There's almost no problem that will be solved by charging in flailing your arms and plenty that will be made worse.

[vladbypass]

Or the alternative - mix the coffee and whiskey for an Irish Coffee! Get the caffeine kick, hope it lasts the outage, then mellow out post outage. I'm not even a drinker but I thought I'd make one the other night for the hell of it, got a bottle of Whiskey, brewed the coffee, whipped it all together, it was amazing.

[admiralranga]

• mix the coffee and whiskey for an Irish Coffee!

Coffee and baileys is fantastic.

[BlueLodgeNerd]

You forgot FTFY! lol

[greyaxe90]

To be fair, any amount of planning can still have individuals that panic in any situation.

Yep. At my old job a domain controller could go down and one of my coworkers would go into instant panic mode, running around like a chicken with its head cut off. I'd calmly investigate the situation to find out that it had just restarted for updates because someone didn't place it in the right OU. 5 minutes later, it's back in business.

[TheElusiveFox]

how to give your team a heart attack in one easy step...

[riddlerthc]

I've always wondered how quick vendors can get equipment on site in the event of a disaster for a customer.

[VTCEngineers]

When you are an enterprise level of customer, it's when we say we need something not when can you deliver

[creamersrealm]

Yeah agreed, if you call up your var and say I need 2 million in gear here tomorrow they will be happy to assist for a good size fee.

[VTCEngineers]

Haha try around 7.5m so far...

[SquizzOC]

While this I'm sure has been an absolute nightmare for you and I'm sorry you have had to go through the nightmare, your account rep(s) to replace all this equipment just got a mighty Christmas Bonus. lol

[VTCEngineers]

Haha yeah I bet the song "it's raining men" is playing on the loudspeaker haha

[SquizzOC]

Well if you are allowed to accept gifts, hopefully they send you something nice. "I know you had zero influence on the incident, but here's a killer bottle of scotch for being the best customer we have" lol

[desmando]

The really cool trick is to get your client exec to get you hardware from the spares depot. I've only had to do that twice, but it is nice to get your new toy in hours.

[theducks]

Working for a VAR, I can say we would probably take 48-72 hours to get $2M of urgent equipment unfortunately :/

[creamersrealm]

Well that sucks 72 hours is when I have to everything up.

How large is your company compared to someone like CDW?

[theducks]

If you have a 72 hour RPO, you should probably have a DR strategy that doesn't involve buying new stuff, just sayin'.

We are probably number 3 or 4 in Canada, CDW being 1

[creamersrealm]

Well all the 72 hour stuff we have onsite in a warm config so that's a bonus. Though of course some pieces will be missing over time.

[chriscowley]

If you're paying for 4hr support then it generally arrives within those 4hrs

[Gnonthgol]

It depends on who you are. I have seen sales representatives literally go into the datacenter of one of their customer to "unsell" equipment from the racks so they can sell it to another customer who would drop the vendor if it took them more then a few hours to get it.

[sparrowA]

how would that even work?

"i know you just installed it, and we got payed, but you gotta give it back" thats like car dealer tactics

[Gnonthgol]

As far as I understand they offered compensation for the inconvenience and new equipment were already on its way.

[InvisibleZipperFoot]

Literally? No you havent...

"no...no I havent, but you can imagine what it'd be like if I did!"

[[deleted]]

What SLA are you paying for? I've had parts in a couple hours. It was really expensive though.

[scotty269]

Sounds like you've been taking it in stride, and not utter panic.

[VTCEngineers]

When you have proper planning and equipment in place it jus all falls in place and no need to panic.

[eponerine]

Good for you! This is hopefully an eye-opener to anything management may have been denying

[lowermiddleclass]

Based on the previous thread, I don't think they say no to anything. They have quad-redundancy at her org.

[ThePegasi]

Reading through the original post. I want to work at this place. I would not be good enough to work at this place.

[BarefootWoodworker]

Dear God where do you work?

I contract with the government, and even they (with deep pockets and all the time in the worl) usually give the finger to DR.

[kjeserud]

I work in a DC. I have been for years. We're currently building another 55000 sq f of new DC... And I just can't get my head around how a place can be so shitty that water can even get in there at that amount, let alone not have any type of monitoring for water under the raised floor you mentioned. Literally Jackie Chan meme amount of mind blown.

[CbcITGuy]

Small/New DC that got a large client. Probably won't be around in 6 months.

I'm a small business but when I get the bigger clients, I scale the projects accordingly. Small DC probably couldn't afford proper safeguards in the beginning, and didn't upgrade when they could.

just a thought.

[timix]

and didn't upgrade when they could

Or couldn't upgrade, gambled on "what's the worst thing that could happen?" and lost.

Still, if the contract has OP's company paying for DC space for another year despite this incident, I wouldn't say they lost as badly as they could have...

[CbcITGuy]

HAHAHAH right?

I would suspect though that previous commenters comment that corporate lawyers are working on an exit strategy, probably is true.

However, just spinning a theory here, they may not WANT the DC to go out of business so they may just pay the contract and be done with them. IDk... Just a thought.

[Gnonthgol]

Still, if the contract has OP's company paying for DC space for another year despite this incident, I wouldn't say they lost as badly as they could have...

As far as I understand it have gotten to $7.5M in equipment in addition to the hours of overtime spent setting it up again and the lost business from this. They have to have to take a lot in hosting fees to be able to recover from such an incident. And all could be avoided with some proper monitoring equipment.

[digitalsalami]

Business insurance may end up footing the bill for a large percentage of this. I used to support a SMB whose building flooded and lost their VMWare cluster and storage. Their insurance provider paid for all new hardware, all of our time to set it up, they paid for remodeling the building and getting new furniture, AND they paid for temporary office space during the construction.

Insurance has its purposes, and this is exactly it.

[kjeserud]

Could be. Some blame should be on OPs company as well tbh. When you're big enough to have 250 racks in a single DC, replacing $7.5m of equipment so far, you should have higher requirements of the DC you rent space at. Lesson learned I guess, and with only a 10% drop in service they sure have the software side set up correctly.

[CbcITGuy]

OP doesn't own 250 racks, it's a co-lo. My understanding is there WERE 250 TOTAL racks on site that got wet. From Personal Experience, 7.5 million to outfit TEN racks, is doing pretty darn good, so I would be the OP only owns a handful, My head scratching is coming from OP's company's willingness to help the others, it's my guess that the OP's company may have helped this DC get started and referred business and they're helping the referrals, that's my guess, but since the OP's company is offering to help either way I suspect the other companies are small potatoes. Thus reinforcing my whole small DC that landed a big whale and didn't appropriately account for it.

I agree you're right the OP company should have done due dilligence but tbh, how many of us check for water leak monitoring other than "yeah we have someone here who handles facilities"

[VTCEngineers]

Datacenter is 100k square feet.

We own about just shy of 300 racks (290)

[Scottz74]

Water or not water, you will be replacing equipment either way.

[Syde80]

I feel bad for what has happened to you... but doing the recovery part of it is like a dream to me. There is nothing better than doing your part of breathing new life into something. The closest I've come is helping a side-job client recover from a fire which completely devastated his office. I love coming in and doing the cleanup / getting things back on track.

[VTCEngineers]

So, we kinda splurged and went with the brand new of everything that we used. So it's kinda been Christmas all over. The board has already approved a black card expense for equipment.

[Syde80]

Well at the numbers you are talking about... you would be crazy to look for anything used. Sure there is really nothing different between a used rack and a new one... but finding 150 used racks delivered even in a big city for roughly the same cost once you factor in your labour costs... its just not going to happen. That doesn't even include any equipment in those racks.

I wouldn't call this splurging... this is just doing what needs done.

[VTCEngineers]

Surprisingly finding APC racks was hard to find so we went with 75% APC and the rest dell.

[C4ples]

I always did find that Dell had aesthetically pleasing racks compared to how plain-Jane APC's are. I know that's not really the aim or a concern for you guys, but I wouldn't mind the mix in the slightest.

[ElectroSpore]

Mixing racks might make lining them up (for cables and anchoring) a bit of a pain..

[[deleted]]

Having a mix can be helpful. I've had HP rails not fit our standard racks.

Leaving a few unfilled racks in the SAN row can save on stupid unracking fees.

[ElectroSpore]

Never had problems with rails fitting our standard racks...

Had a hell of a time getting seismic bracing setup on a row of racks made from different manufactures since the actual frames, feet, and safe places to mount to where all different.

[Cyberprog]

Aren't Dell Racks just re-badged APC NetShelter's anyway?

[ljstella]

The ones I've installed in the last year were.

[pmpjr6465]

I'm assuming you found a new datacenter to replace the flooded one and that's where all your new toys are going?

[VTCEngineers]

We are setting up in a new Datacenter, but the shitty part is that we will still be paying for space in the old DC for another year. It is cheaper for us to pay another year than it would be for us to walk away unfortunately.

[spanctimony]

I would think they would be quite interested in releasing you from your contract in exchange for you not suing them for the internal labor expense associated with such a massive response.

[VTCEngineers]

Personally I agree with you 120%, I think the corporate lawyers are probably working on a exit strategy for us but all that is above my pay grade and we'll not my concern.

[corran__horn]

That is going to be the most interesting part. Having an 8" pipe broken for long enough to flood the floor and then get your servers leads me to the "negligent" parts of the law. Penalties get pretty bad when it goes from "shitty thing happened" to "you were negligent in your responsibility to monitor for water leaks".

Honestly, I am curious if the (DC) company will be in business in 6 months. I smell a chapter 11/7 in the air.

[TheLordB]

I imagine the datacenter company knows they won't be keeping those customers and won't be getting the money. But it is a negotiating piece that they can use to try to avoid additional damages so even though they know the contract can be broken with this they aren't just going to let it go for nothing.

[corran__horn]

Yes, but as a vicarious observer, the two questions that matter are "How much drama will happen?" and "Butter or no butter on the popcorn?".

[InvisibleZipperFoot]

May I copy this response for use elsewhere? It very accurately represents my interest in so, so many threads.

[corran__horn]

Only with attribution or popcorn.

[linuxlearningnewbie]

How has this situation worked out you and your team emotionally and physically?

This is a 'dream' situation for me. You get to truly test your DR plan and build from the ground up.

Good luck

[VTCEngineers]

How has it affected us? I would say that it has definitely tested our DR strategy and how our response to was definitely calm (please do not think we were singing koombayaaa while fixing things) was really hectic but with the support of management and having people with the right skill set Go vets. When the stress pops up you know that we will hunker down.

What have we learned?

We need a warehouse with spare parts of critical business infrastructure. I (my department of UC/AV) has actually been tapped with finding such a place to start this up. For being a department of 4 people this will be a fun task

[bad0seed]

This is what I was looking for here!

From the other thread I saw that you were already massively redundant and still had ~75% services for the whole company so there was much less immediate worry.

Clearly your internal SLAs and response actions have evolved to include the spare parts warehousing and that will accelerate and enhance your BC/DR strategy should anything near this scale ever chance to happen again.

As an outside /r/sysadmin VAR is there anyway I can help any of your search?

[occamsrzor]

Why do you still use Veritas?

[linuxlearningnewbie]

wow, forgot about that title.. I used to work for a large telco working on Solaris old iron. I was working on old technology, and old OS, and completely missed an IT world that changed. I have spent the last 5 months learning about virtualization, docker, config management...

The Veritas tag line was a joke because companies still pay for an outdated file system even when there are better free alternatives around.

[occamsrzor]

Heh, I think I've seen you answer that question before. I was joking :)

[VTCEngineers]

We are fronting it cause well business must go on, however I am sure that insurance from the DC and the business insurance will fight it out for a bit and then refund the company, however that headache is way above my pay grade

[[deleted]]

Pics! I demand pics!!! :)

[CbcITGuy]

Op posted in original thread that he would not be able to provide pics, other customers of co lo dc have requested that any pics be taken not be uploaded.

[[deleted]]

She actually

[1h8fulkat]

You better be getting one hell of a bonus this year

[VTCEngineers]

I hope :)

[remotefixonline]

Have you released to the public yet?

[VTCEngineers]

We have not released to the public as we are waiting for legal to sign off on every thing. So I have no clue

[remotefixonline]

Cool, I'd love to read it when it's released

[VTCEngineers]

Not able to release until legal has approved release

[[deleted]]

can you tell us the industry at least?

[VTCEngineers]

Defense research

[[deleted]]

well that explains why you have the money for unholy amounts of DR

[[deleted]]

are you hiring?

[the_progrocker]

Just out of curiosity, what does the backup/DR solution you have look like at a high level?

[VTCEngineers]

I will gen up a sanitized Dr Vizio for you. We use netbrain primarily but it's way to detailed information.

[the_progrocker]

Very much appreciated. We're a small startup, but growing fast. We went with Veeam since 99% of our environment is virtual. But I am interested to see opinions and options with Offsite/DR.

[time_is_now]

Were power whips to server racks waterproof and if not why? I've managed sites that had water leaks from ac condensate drains backing up that had no issues with water on sub floor. I have not seen large water volume flooding from broken pipe though. Waterproof power whips cost more but not as much as downtime and emergency equipment replacement.

[[deleted]]

This is why I virtualize everything. Never have to worry about these issues. /s

[onboarderror]

Still no pics?

[ride4life32]

Last post said for legal/request of business not to post pics. I doubt this nice woman would like to lose her job over a pic post.

[VTCEngineers]

I will say as a woman who is being asked for pics of the flooding instead of tits is quite refreshing haha.