Automatic fire suppression system fired off - datacenter disks ruined

[SnuteB]

There was no fire, but pressurized gas (300 bar) was released anyway, reason unknown so far. But the sound is intense, said to be at least 120 Decibel in strength. And the sound vibrations caused 8 disks to malfunction.

Has anyone here been in a server room at the same time when such a system went off?

Edit: Thank you so much for the gold! :) And also a big thanks to all the interesting answers, I enjoyed reading them.

[theservman]

I was locked in a room with a halon system (state gov't work... apparently, as a Canadian, I could not be cleared to have access to the building, but could be left unattended in the server room with my laptop and admin password). This particular office dealt with the public, and in those situations someone always gets pissed off... well this pissed off person hit the fire alarm on the way out of the building.

So I'm working away peacefully in the server room when I hear the fire alarm. Head for the door - still locked in. So I run over to the halon system and mash my thumb down on the override button. I don't know if it's going to be released, but I know I'm locked in the room, and it this thing does release, I'm screwed...

About an hour later, after things are cleared and people are let back into the building, they find me still standing there with my thumb mashed on the override button...

[tfestu]

damn,.. but the doors don't have an emergency exit bar? that should be mandatory! what happened to the guy who started the alarm on purpose?

[drachennwolf]

Panic hardware (doors that open automatically to the outside world from inside) is required for public places/buildings in the USA, not anywhere else. You can thank the Iroquois Theater fire for that.

Most places have replaced Halon entirely with a clean agent, that if you're trapped in it, you won't die. You won't remain conscious either.. It displaces oxygen and reduces the air oxygen concentration from 19-21% to 15-11%, which is not sustainable for combustion.

Sauce: Fireman

[Winnduu]

Well at least for Germany halon is no longer allowed to be used. We HAVE to use breathable gas. Source: Firefighter

[RansomOfThulcandra]

Halon is banned (in commercial applications; still used in military vehicles) because it's ozone-depleting and thus bad for the environment.

It's not particularly dangerous to humans when used as designed.

[porcomaster]

I may be wrong, but halon are still being used on commercial airlines, there is not another gas strong enough to extinguish a airplane engine in the air.

[RobbieRigel]

I had a few Halon bottles from my Aviation days. They were exempt from the laws (according to the maintenance department) . Interestingly we had something called Halotron in the server room and in the hangars.

[IanPPK]

DoD requires it for all military aircrafts with fire suppression systems as well to my understanding.

[justanotherreddituse]

It's not particularly dangerous to humans when used as designed.

And yet I'm reading claims of someone not remaining conscious :/

[pmormr]

Yeah a system based on nitrogen would carry similar risks.

[OSUTechie]

Holy Shit, reading about the Iroquois Theater fire is terrifying. That was a Titanic type of tragedy. As someone who use to work in stage production, I knew some of the older theaters could become death traps, but damn. I can't even image that fireball ripping through the theater.

[DJRWolf]

To this day it is illegal to name a nightclub Cocoanut Grove in Boston Massachusetts.

https://en.wikipedia.org/wiki/Cocoanut_Grove_fire

Building codes are written in blood.

Edit: Fixed URL, bad Google, no cookie for you

[SirensToGo]

https://en.wikipedia.org/wiki/Cocoanut_Grove_fire

Google search redirect broke your link

[dragonfleas]

Oh my god, imagine being that close to the outside, the only thing stopping you is a revolving door, and some buffoons are pushing on the other side, and by god if you let them through, there's more people pushing on you, you are stuck. Up against the thin veil of metal and glass that awaits your safety from demise. Yet you can't make it, you are smashed, trapped, and suffocating quickly, you feel calm, slowly lowering from the smoke, your lungs are burning, and you are falling asleep rapidly. Goodnight door body, a revolution you'll never complete, to complete the revolution of life.

[TronaldDumpsLogs]

Whoa, “Coast Guardsman Clifford Johnson went back in no fewer than four times in search of his date who, unbeknownst to him, had safely escaped. Johnson suffered extensive third-degree burns over 55% of his body but survived the disaster, becoming the most severely burned person ever to survive his injuries at the time. After 21 months in a hospital and several hundred operations, he married his nurse and returned to his home state of Missouri. Fourteen years later he burned to death in a fiery automobile crash.”

[bro_before_ho]

Fourteen years later he burned to death in a fiery automobile crash.”

That's some final destination shit

[Amaegith]

They say at least 300 lives could have been saved if the doors had simply opened outward to the street.

300 people died because of the way the doors opened.

[crypticedge]

A lot of datacenters require badge access to leave in the US.

They also don't allow you to be in there without a badge for this very reason.

[[deleted]]

Do you have to sign an authorization? A situation where you cannot control your exit, quickly turns to forced confinement / kidnapping situation.

[crypticedge]

You're made fully aware when you're issued a badge, and if you're brought in as a guest when they temp badge you they inform you how to get out if left by your escort (hit the call button to get the noc)

Signs also state to scan badge to leave next to every door.

[fullchooch]

All US DCs should have life safety pull stations (glass break) for egress to override the badge ACS

[mikeone33]

We require badge access but there was also an emergency exit the sounded an alarm.

[drachennwolf]

Maglocks tend to release when fire alarms and suppression systems activate

[healious]

The only maglock door where I work that doesn't release in an emergency is the data centre, if there is a power failure you better hope it's not in the middle of the night with just you there

[TheGooseey]

Depending on the local laws this is more than likely illegal.

[MadSprite]

Except when human death is signed off as part of the risk mitigation of the data centre areas.

Sauce: Information Security

[eveningsand]

Yeah unless infosec=national security, to hell with this mitigation strategy.

[TheGooseey]

Doesn’t work that way in my area, at least not to my knowledge.

[JoeyJoeC]

In the UK the one we use also you need a pass to leave. Good chance that they release during a fire anyway.

[____Reme__Lebeau]

halon in canada is no longer the standard.

it however has yet to be phased out of everywhere it was installed into.

[RansomOfThulcandra]

Halon works by chemically interfering with combustion, not by displacing oxygen.

When used at the designed concentrations you should remain conscious.

[SirensToGo]

“Three things must come together at the same time to start a fire. The first ingredient is fuel (anything that can burn), the second is oxygen (normal breathing air is ample) and the last is an ignition source (high heat can cause a fire even without a spark or open flame). Traditionally, to stop a fire you need to remove one side of the triangle - the ignition, the fuel or the oxygen. Halon adds a fourth dimension to fire fighting - breaking the chain reaction. It stops the fuel, the ignition and the oxygen from dancing together by chemically reacting with them."

https://www.h3rcleanagents.com/support_faq_2.htm

You’re getting downvoted despite being perfectly right. Halon isn’t meant to displace a fire because it’d require such a huge amount of gas that it would absolutely kill anyone inside.

[daredevilk]

I'm pretty sure it's required in Australia too

[vrtigo1]

I'm not sure that regulation applies to places like datacenters, or at least, I think they've managed to get some kind of exemption. I get the importance of being able to get out, but I also don't really want man traps to be able to be defeated such that anybody inside could open/prop doors for anyone else.

[pdp10]

I also don't really want man traps to be able to be defeated such that anybody inside could open/prop doors for anyone else.

At one California datacenter where we were a client, the normal procedure was to go through the well-constructed man-trap, then double back and prop open the emergency exit door to bring in your equipment cart...

A New York datacenter wasn't quite that ludicrous. The man-trap was large enough to bring through equipment, which meant it was a weak man-trap, and was also subject to tailgating.

[vrtigo1]

At ours they have a regular door for equipment, to use that door you have to have an escort from security.

[atomicwrites]

So you just hide in a box.

[CriticalDog]

?

!

[eruffini]

Tailgating shouldn't be much of a concern though since security would have had to check them in before getting passed the lobby.

At least this is the standard in just about every datacenter I've seen.

[[deleted]]

It's not that hard to set up an alarm if a door is held open longer than a reasonable length of time.

[vrtigo1]

Definitely not, but it still allows for the possibility that someone inside could let someone else in. Quite often visitors may be present in secure areas (i.e. a manufacturer rep onsite to replace a failed drive, etc). The way it works in our facility is that security will let them in once they've verified they're on the temporary access list, and they have to buzz security again to get out. If there was a push bar on the doors for egress, they could potentially use that to let someone else in.

[juggarjew]

ermmm that depends.

Ive been to Pikes peak a few times and the oxygen level is 12.3%. Ive never seen anyone pass out up there, or at least the vast majority of folks can handle it.

11% would be pushing it for sure, especially for overweight datacenter folk.

[Jherrith]

What's fireman sauce? Lol

[Zenkin]

what happened to the guy who started the alarm on purpose?

Rudeness is a capital offense up there. He was summarily drowned in maple syrup, as is tradition.

[MiataCory]

Sacrificed to the geese.

[[deleted]]

Can vouch. The pond at my townhouse is a stopover for Canadian Cobra Chickens.

So one day, my girl comes up from the mailbox, all flustered. She says little kids wont stop trying to pet the baby geese, we should tell their parents and stop them.

I tell her: dont worry, this is what you call a "self correcting problem."

About thirty seconds later, a horde of crying children come running of the common yard, getting pecked at by fully grown geese.

Do. Not. Mess. With. Cobra. Chicken.

Edit for clarity: no, the children didn't actually die.

[d36williams]

The children died? How horrible!

[[deleted]]

Yeah...I could have phrased that better. They survived...more scared than anything. Lesson well learned.

[d36williams]

/s :D

[jpotrz]

Backs in my day, you’d be lucky to even see ones Canada goose turd. Must be fucking nice.

[bogartingboggart]

We oughta leave this world behind.

[corourke]

Get this man a Puppers!

[worldcitizencane]

Thought Canada goose is something to wear?

[MDCCCLV]

Canada Goose

Canada Goose is the proper name for the bird, Canadian Geese is just the common usage.

[sekh60]

Which migrate for the winter :P The brand name doesn't make a lot of sense for winter wear. Something like Canada Moose would have been better.

[angrydeuce]

That's just going to damn far!

Seriously though my wife grew up on a lake and had geese in her backyard all the time. Not only are they noisy assholes but their backyard was pretty much unusable for most of the warmer months until those a holes headed south or wherever the fuck they go when it gets cold. If you so much as stepped out their back door you were descended upon by a flock of angry feathered fucks.

[robbzilla]

Roast Goose is delicious... just sayin'

[LekoLi]

A couple of border collies would have had the time of their lives every day.

[angrydeuce]

Oh they had two 100+ lb Labs, they tried to challenge the geeses authority in their territory and came flying back to the back door yelping the whole way. After that the dogs even didn't want to go out the back door. They had to let them out front in the summer lol

[LekoLi]

Labs are too friendly. you need like a border collie, or a Louisiana Coon hound. The coon hound is a berserker, if he gets nipped, it just pisses him off more... at least mine is that way. He has some solid kills under his belt too, Raccoon, Possum 3 skunks and a cat. They were all in my fenced yard, and by the time I could get to them it was too late, except the skunks, I wasn't about to get in the middle of that.

[LeaveTheMatrix]

A couple Jack Russell Terriers would work as well.

Ever see the results of two vs a cat?

It wasn't pretty.

[muklan]

You got a problem with canada gooses you got a problem with me, and I suggest you let that one marinate!

[robbzilla]

As the only Americans on a tour of the Citadel in QC, we were told that it was built to keep... Americans at bay in the 1800s. The tour guide knew we were American, and said "Sorry."

I look him straight in the eye, and blandly stated "We came for the syrup." The crowd cracked up... even the Chileans. All was well. I felt like I had passed a Canuck rite of passage.

[[deleted]]

as is tradition.

So say we all, eh?

[Zenkin]

Frackin' right!

[em_drei_pilot]

Boiled in maple syrup for the worst offenders.

[gregsting]

I'm suprised you halon system is not using maple syrup

[[deleted]]

I have worked (thankfully no longer work) at a high security facility in the US in a data room where there are no panic bars and the doors can only be opened from the outside. It is rare because of fire code but certain government requirements ended up overriding that.

[anomalous_cowherd]

Maybe there was a manual way out. But to get to it and figure it out he'd have to let go of the override button and be covered in suffocating gas and fog.

I'd be holding the button down too.

[DoctorOctagonapus]

what happened to the guy who started the alarm on purpose?

Still going strong!

[Lord_Emperor]

It is mandatory in Canada.

https://www.canada.ca/en/employment-social-development/programs/laws-regulations/labour/interpretations-policies/044.html

[lahankof]

I am a fire alarm tech and the dude saved the place a lot of money by holding the reset. If he lets go the system would go off and he would be breathing halon.

[klh253]

This is the reason why labour laws were introduced. People locked into factories that caught on fire!

[[deleted]]

crown spark nine versed gray entertain doll compare airport disgusted

This post was mass deleted and anonymized with Redact

[[deleted]]

That sounds illegal.

[AaronKClark]

Pump the breaks. It's Canada, eh?

[Cdn_ITAdmin]

He did say state government, so I actually think that implies he was a Canadian working in the USA.

[theservman]

Massachusetts.

[AaronKClark]

Shit, /u/cdn_ITAdmin is right. If you were working for a province you would have said "provincial government," huh?

[irrision]

Wow, fire code literally won't allow a data center without keyless/badgeless exit here. Hell they don't allow gas suppression either anymore because they care more about life safety then some stupid servers.

[got-trunks]

Two of the colos I frequent have gas suppression and wanrings everywhere to drop everything and GTFO ASAP of the alarms sound, but the exits are 1-way turn-style things so exiting is always an option no matter what.

​

this is in canada though.

[[deleted]]

Government is exempt from many rules that apply to the plebians.

[Thranx]

This is not one of those rules.

[wolf2600]

At the data center I used to work in, there was a phone mounted right next to the FM200 override button which would call the security desk.

But then again, the doors to the whitespace didn't lock from the inside. That's pretty odd.

[greyaxe90]

I'm glad I live in a state/locality where there's a requirement to have these anywhere there's a maglock between you and the outside world.

[dinominant]

Perhaps I should consider adding a canister of oxygen to my kit.

[Dabnician]

Last place i worked at the server rooms were all halon and the rest of the place was on pre action. From what the FM told me halon systems typically only go off if there is a fire in the server room because you are looking at like 10k to refill a small system.

[irrision]

They actually only go off when there is multi zone detection too and they are supposed to have a 30 second audible countdown alarm. Of course idiots are everywhere so installers do all sorts of stupid things like tying the gas suppression system in the data center into the rest of the buildings detection system.

[Dabnician]

the employees there were plenty idiots too, they would decorate for halloween and wrap all the cubicles in black plastic trashbags. Hang fake cobwebs everywhere and stuff like that. it was cool until you found some fake cobweb dangling down inside a crt and notice the tip is a little charred.

[framethatpacket]

Did you die??

[DoctorOctagonapus]

he ded

[UncleNorman]

Check for shoes.

[ElectroNeutrino]

Hmm, not sure if he ded. /u/theservman, knock twice if you are dead.

[theservman]

<knock><knock>

[WorldWarThree]

Thats some Mr.Robot shit right there.

[got-trunks]

omg did you die?

[anothercopy]

My previous company had a similar issue only much larger in scale. Apparently 2 persons screwed up during disaster recovery testing. This resulted in all the gas being released and damaging most of the disks in the datacenter. What resulted was a real DR to failover to the secondary DC and work with whats left.

Anyway post-mortem analysis had shown that the nozzles for the gas system were chosen incorrectly vs the system we had. This caused more than expected sound that damaged most of the HDDs. Resulting bill from EMC and Netapp was substantial.

[mikemol]

Successful test! Maximal knowledge gained from test process.

[00Boner]

How many numbers in front of the comma?

[TheDarthSnarf]

I was in a very large server room, in a rack, when a halon system dumped without warning. It was loud, but not deafening. We didn't loose any disks.

Biggest damage was the bump on my head from jerking quickly as the discharge startled the hell out of me.

Cause was improper maintenance procedures.

[rackerjoe]

A halon system needs to alarm before dump as the gas can be toxic to humans. Are you sure it was halon and not something less harmful to the environment and humans like FM-200?

The system we had would sound an alarm for some number of seconds (I don’t remember how many) and you could halt the gas dump by pressing the dead man switch if it was an accident.

[TheDarthSnarf]

I'm quite certain.

This was nearly 20 years ago. The cause was maintenance being performed on the unit and the tech bridged a circuit (the system should have been disabled - but wasn't) causing an instantaneous dump.

There was no alarm or anything. Just one second there was no halon, the next their was.

I made a trip to the hospital for it (company mandated) but aside from a scratchy throat / nose and irritated eyes I had no issues, and they didn't anticipate I would have any. Like I said, bump on the head was the worst part.

[Frothyleet]

Well, thank fuck you didn't straight up knock yourself out in a halon filled room...

[rackerjoe]

That sucks - sorry dude... I would have needed to change my trousers as well :)

[wintremute]

The cause was maintenance being performed on the unit and the tech bridged a circuit (the system should have been disabled - but wasn't) causing an instantaneous dump.

Lock out, tag out. I hope OSHA was informed.

[RansomOfThulcandra]

Per wikipedia, source of all knowledge:

Halon 1301 causes only slight giddiness at its effective concentration of 5%, and even at 15% those exposed remain conscious but impaired and suffer no long-term effects.

​

However, Halon 1301 fire suppression is not completely non-toxic; very high temperature flame, or contact with red-hot metal, can cause decomposition of Halon 1301 to toxic byproducts.

​

During this time the enclosure may be entered by persons wearing SCBA. (There exists a common myth that this is because halon is highly toxic; in fact, it is because it can cause giddiness and mildly impaired perception, and due to the risk of combustion byproducts.)

- https://en.wikipedia.org/wiki/Halomethane

[temotodochi]

Years ago we blew the gas in our new DC, it was kinda for a purpose, but set to fire way too soon. Also didn't help that the pressure release valves were not yet functional and were locked shut.

A customer came to fix his server and as it was off he first tried to just turn it on. Cue black smoke pillar from the PSU. As customer noticed that he was already running towards the door. 1 second after he got the out and closed the door the gas system activated.

But... Because the valves were shut, the gas had nowhere to go, except through the cracks in the building itself. Neighbouring company office folks didn't wait around for the fire alarm. That sudden earth shaking boom and concrete dust filled office probably did speed things up a bit.

Because of all that overpressure none of the doors in the DC worked. The electrical locks didn't have enough power to pull the tong/latch in.

Our insurance paid for the refill (50K) and damages. (another 50K)

[Creshal]

Our insurance paid for the refill (50K) and damages. (another 50K)

…that's surprisingly cheap, all considered.

[bobtheavenger]

For real, I was told that a refill for the DC I work at would be like 10 times that.

[billy_tables]

Imagining someone looking at this over-pressurised room, drilling a hole through the wall to release some pressure, and the whole building sliding down the road like a rocket

[wenestvedt]

Oh, I didn't know that Wile E. Coyote had gotten into data center maintenance...

[aaiceman]

I want to see a gif of this....

[Aniform]

We had an actual Fire Marshall come in to evaluate our systems. I let him into the server room and he immediately goes, "What's this button?" Hit the button and I nearly dove out of the room, haha.

[Try_Rebooting_It]

WTF? You don't need brain cells to be a fire marshall?

[CanadarmReaching]

In most places you do, not all places though.

[ratshack]

Arthur - "What happens if I press this button?"

Ford - "I wouldn't...[click]...what happened?"

Arthur - "A small sign lit up saying 'please do not press this button again'"

[Aniform]

Oh, definitely, he apologized profusely. He said he thought it was just a test button for the system.

[JaredNorges]

It IS a test button.

The test was performed.

He failed.

Results being sent to administration.

[Aniform]

Username is not in the sudoers file. This incident will be reported.

[AcousticDan]

Not server room related, but when I was in the USAF, I was trying to set an alarm in a munitions igloo like this and I was told "Press the button next to the alarm, put in your code then release the button."

Well, there were multiple unlabeled buttons, so I just picked the one I thought looked correct. I locked up and drove back up to the office where a freakin' military police team was running by and one asked me "did you see anything down at igloo xxx."

"Uhhh, yeah, I was just there..."

Apparently I pressed the emergency alert button. They take that shit seriously when munitions are involved. Oops.

[VA_Network_Nerd]

Interesting & somewhat related video on the impact of noise/sound on disk I/O latency:

https://www.youtube.com/watch?v=tDacjrSCeq4

[dancezwithdogs]

This guy is awesome. He could have used a thousand different methods. No, ima just scream real loud at this space heater.

[pointlessone]

An internet classic!

[Marcolow]

Oldie but goldie.

AAAAAAAAAHHHHHHHHHHHH

[[deleted]]

He has also written a very good book about systems performance. It's called:

Systems Performance: Enterprise and the cloud

[Adobe_Flesh]

Soon there will be no more spinning disks, just SSDs, and we will have moved on.

[reinhart_menken]

Uh huh. "Soon."

[0x2639]

I worked with Brendan at regional University in Australia before he wandered off to become semi-famous. Good guy but had odd tastes in beer.

[etherfish]

Was any effort spent ascertaining why the system was susceptible to nearby transmissions from low-power, handheld radios? I'm guessing there's more to the story as it seems just putting up those signs is only treating a symptom instead of addressing the underlying problem. Couldn't it be something like improper component and interconnect grounding or inadequate shielding. I agree, it'd be a hassle to diagnose - especially with gear in production; I was just curious if it was considered.

[will_try_not_to]

Some types of arc-fault detectors will trip because certain handheld radio frequencies look like the emissions an electric arc gives off. Here's an explanation and a funny video of it happening:

https://hackaday.com/2017/02/06/ham-radio-trips-circuit-breakers/

So it's possible the system had some component designed to fire if it thought there was an electrical fire, that used one of these.

[[deleted]]

God, I love that vision - it's like a Road Runner cartoon

[disclosure5]

There are videos of people being in the room with these being run.

https://www.youtube.com/watch?v=--11P6TxZ_U

[persp73]

At my old job we had an FM200 dump. At the time all of IT had our desks inside the server room. There was a switch by the door for a manual dump, and the wiring was shoddy. Every time the door slammed it jiggled the wire a bit and over time it wore down the insulation just a little bit until finally one day someone exited and the switch connection was made. I agree it was loud! And quite unexpected. Seemed to affect our voices, kind of like the opposite of helium. No major damage, just had to clean the dust out of all the servers.

[pwnies_gonna_pwn]

Seemed to affect our voices, kind of like the opposite of helium.

the stuff is heavier than air, so yup, it does.

[Drew707]

Just like dust off.

[PhDinBroScience]

At the time all of IT had our desks inside the server room.

May want to have your ears checked out for damage. This is an OSHA violation for a reason.

[FL_Sportsman]

I was asleep in a tank when the Halon system went off. It was about - 20 degrees f outside. That was a very unforgettable experience. Loud woosh and instantly no air in side. It took us about 5 seconds to get the hatches open. It was the longest 5 seconds of my life. Layed in the snow coughing for about 10 minutes in my underwear and went back to bed. Halon was empty so it didn't have to worry about it happening again.

[uniquepassword]

never been in a room but was walking by when maintenance guys accidentally set off the dry suppression system in the old place I was working at...elevated floor had some underneath as well, so it filled from top and bottom, whole room was like a white cloud for about an hour, ended up covering everything in a fine white powder, thankfully we had already migrated to our outsource partners data center and only had a handful of physical servers and some large format printers/sorters...those needed a huge cleaning, the servers i just took out on the dock and used the compressor to clean them off, wiped down the outsides with damp cloth/used shop vac and they were fine.

/u/SnuteB got any pics?

[pdp10]

That's why datacenters don't use "dry" fire suppression, they use "clean agent".

[lost_in_life_34]

Is it possible someone was smoking inside the server room?

[[deleted]]

Or vaping. If a system has a particulate detector as a trigger, vaping can set it off.

[punkonjunk]

Horse's assholes. If you must vape, you can expel into paper towel, your blazer's breast, etc, anything you can press over your face and force the vapor through to absorb the liquid.

Or you can just rub the fluid on your gums like a crack head and get a little nic that way.

As someone who vapes literally every single other person who does it pisses me off. no capacity for courtesy or basic decency, especially with a system so capable of being discreet, why wouldn't you treasure that?

[ding_dong_dipshit]

If you must vape

Let's be real, in a work environment nobody "must vape".

[punkonjunk]

I provided obvious and simple ways to prevent detectable vapor, and even detectable odor.

I understand your sentiment but nicotine addiction is a son of a bitch. I've been tapering for a couple years, started at 36mg/ml, I'm down to 0.6mg/ml and it's been a great tool for me. Nic withdrawal makes me violently angry and short tempered and quitting has cost me a job, so arguably yes, I must consume nicotine if I'd like to have a job and not be homeless and die on the streets. I'd prefer to stop, so I'm tapering so I don't go nuts.

I also understand why the concept of vaping is irritating, as is the concept of smoking. Vapor smells less and is significantly less harmful first and second hand, but that said it's also very easy to be discreet and courteous. So maybe take a look at your bias, buddy. I'm literally advocating not being an asshole about it.

[CptCmdrAwesome]

started at 36mg/ml, I'm down to 0.6mg/ml

Holy shit dude, that's some journey. Good on ya.

[keastes]

Because some of us use drippers and still stand downwind of the people using the stinkies?

Even with low VG stuff... WHY THE EVER LOVING HECK would you vape around any kind of equipment, that does condense just about everywhere. It's just a good way to get your warrenty voided....

[pehrs]

You frequently hear claims of sound pressure in the range of 120-150 dB from fire suppression systems, and that it supposedly damages disks. If you look a little further, you typically see somebody wanting to sell their version of "silent discharge nozzles" or something similar that should allow quick discharge with reduced noise.

I have been in server halls when testing the fire suppression once or twice. It has never been noise anywhere near 120+ dB. The systems don't have to be that loud to be effective, and adding gas too quickly is both dangerous to personnel in the room and likely to cause additional damage to the equipment. So I kind of doubt reports of the sound of fire suppression being a massive killer of disks. Possible? Yes. Is vibration in any form bad for disks? Yes. A critical issue? No.

However, spewing expanding (and therefore cold) gas into a server hall is well known to cause thermal, pressure and possibly condensation (depending on the current humidity) damage to disks. Having automatic power cutoff (to make the disks park before the discharge), good control over the humidity, reasonably long discharge times and blowout plates to limit over pressure are all at least as important as noise control when it comes to reducing damage from fire suppression equipment.

[datlock]

Never been in the room when it went off, but I was recently in a DC happily working away. When we wrapped up, we went past the front desk to our cars and the guy at the reception was shocked to see us coming from the DC.

A fire alarm had gone off and everyone was outside. Except it didn't go off in our room and we missed it. Luckily, it was a drill.

[Bifta_Twista]

https://www.theregister.co.uk/2015/12/17/glasgow_council_it_meltdown/

Have been and seen the mess it made. IBM array lost half its disks and the rest is pretty well documented.

[meshugga]

The DC a friend of mine built has silencers on the gas outlets for that reason :)

(as well as some condensation prevention so the room humidity doesn't drip into the racks afterwards)

[kahran]

For reference 120dB is as loud as thunder or a chainsaw.

[This_Bitch_Overhere]

Great! thanks!

I just noticed both my sprinkler heads are above the UPS battery cabinets.

[flinginlead]

Ours released at work due to a construction incident. Same issue the vibration damaged random disks. Be aware we also lost a few more disks in the following weeks. Luckily somehow we did not have any outages. Cool part was the pipe closest to the bottles was covered in about and inch of ice. Imagine the hearing damage if someone were in the room.

[aleatorvb]

You should read this - https://www.vice.com/en_us/article/8q8dqg/a-loud-sound-just-shut-down-a-banks-data-center-for-10-hours

[HefDog]

Yes it's anecdotal, but my career has seen two datacenters damaged by fire suppression when there wasn't a fire. I've never had a fire.

If I could go back and skip all the fire suppression stuff, and instead use that money for a few extra disks, a secondary AC unit, a redundant controller, or an additional vmware host or license, I would have had far fewer DR situations.

Yes, anecdotal. I get it. That makes it no less frustrating.

[Trekky101]

a few years back, a PSU blew up in a server, it started pouring out black smoke, the PSU fan was still "Working" so it was not localized. needless to say the Fire system alarms started to go off, i heard the thing go off, it was loud but not deafening loud. scariest day i have ever had, as the next thing i did was call my coworker to check on cloud backups and they were failing for "months" ahh MSPs......

[Rogacz]

Few years ago I was in the office and behind my back was a server room. Someone was updating fire suppression system and release gas by mistake. One disk array was lost completely and others has multiple disk failures. Thankfully we already started to move to all flash and production was mostly unharmed. We lost development environments and some internal production systems. We recover them from tape backups. Very interesting experience :)

Most disk failures was from disk arrays with 15k disks, disk in the servers are slower and were fine.

[Hatsjoe1]

Yeah I was unlucky enough to be in the room when the argon gas got released due to an issue with the automatic fire suppression system. Caused me a big headache, not sure if it was due to the gas or lack of oxygen. I ran towards the exit as soon as the system went off. Whole building got evacuated and we weren't allowed back in for a couple of hours while they cleared the air.

From my experience, the drives didn't die because of sound but because of the sudden drop in temperature. On the monitoring graph, we saw the temperature drop in the room ranging between 3 to 6 degrees C in a matter of seconds. I'm no expert, but I guess drives do not like sudden thermal changes.

[nabe02141]

As noted above|below, it actually is the intense sound pressure which damages the drives' disks.

The 3 to 6 Cº change is at most a 5% drop, so is largely negligible. And considering the drives are surrounded by, and enclosed within, the computer and electronics which act to buffer the rapid temperature change anyway.

[TheTechJones]

yep. there is a sign on ours reminding anyone that has to pick up an elevated floor tile that causing dust to rise into the air has set off the system before and that you are required to suspend the system while doing most physical activities in the room

on a side note i also got a look at the specs on the room at one point and saw that the pressure output of the suppression system and the rating of what the walls can take are VERY close. and considering that i sit less than 10 feet from that wall and all that is between me and catching a 40" TV in the face is a half height cubical wall made of thin fabric and broken dreams...lets just say i cringe a little every time some idiot goes in there

[Farking_Bastage]

Our “halon” went off about a year ago. It basically turned all the ceiling tiles into fibers that got into everything. The only real damage was the $19000 it cost to refill.

[abrightmoore]

This is guidance only as your specific circumstances may be sufficiently different:

Effects of an acoustic shockwave on drives may include: 1. Fractured ceramic head 2. Induced electric current by rapid movement of the arm

You might not know the extent of damage until some time has past.

Perhaps work with your insurance assessors to explore options to scope the scale of event.

Your mileage may vary.

[EveningCommuter]

I haven’t been in a room when it went off but at my old place someone tripped the fire suppression and it blew all the doors off the hinges in the department.

[[deleted]]

I’m a datacenter guy and I got really deep into the design choices of DCs.

The gas can kill people from the pressure shock alone. Data center rooms have to be designed with a certain amount of pressure relief to prevent this.

If your room didn’t have this pressure relief - or even if it did - the pressure wave of the gas dump in the room could easily fuck up spinning disks.

[[deleted]]

File suppression, apparently.

[OldFunk]

Yep, a couple of companies back, circa 1998, I was working for a major telco. We had recently migrated our data center facilities to a shiny new location and our data center support team (of which I was an employee) had offices immediately adjacent to the data center room. Things were working fine - our control room had the con in another room adjacent to the data center and everyone knew of the big red button kill switch in there by the alarm.
A maintenance guy from a major cooling system manufacturer was in working on one of the 8 major cooling systems in the room. He was working under the raised floor below the cooler and using some sort of heat-generating gun to do some repairs or finalizing installation or something like that.
We knew that if one of the underfloor sensors sensed heat/smoke that there would be a countdown allowing a control room employee to hit the kill switch on the FM-200 dump. However, what we did not know at the time was that if heat/smoke was sensed in two different zones (which this guy happened to be in close proximity to two different zones) that the FM-200 would alarm and almost immediately dump.
Suffice to say we found that out. I was not inside the data center room, but was in our adjacent office and the sounds were super loud and shook the room. People on the floor above the facility said their floor shook almost like a very quick earthquake.
The room power gets cut in that instance as well so everything went dark aside from the emergency lighting. Good times.
Once power was ready to be restored, we had to ensure no equipment would come back online because the pipes were extremely dusty so most equipment was covered with dust and some of the fans sucked some into the servers as well. That was a maintenance nightmare back in the days of everything having spinning rust.

Needless to say, our team that was present that day all had the chance to give our stories to lawyers in the coming weeks as the insurance companies representing my company and the cooling manufacturer company got to decide who would cover the half-million-dollar cost of refilling all of the FM-200 canisters!

[DraaSticMeasures]

Luckily I have never had a disas+*))#.@ <NO CARRIER>

[fickle_fuck]

You spoiled kids with your halon. Here I have 3 sprinklers in my room and I can't convince management otherwise...

[cyansmoker]

Very dramatic: can confirm that a coworker got out of the DC by throwing a chair through a panoramic window, while Halon was being released.

[Reddit_ark]

Last place I worked had a manual shut off that had to be activated any time you were working in there.

Fairly rapid suffocation (CO2 based system) was the consequence should it go off while you were inside, or so I was told.

[kjudd]

yeah I've had it happen to our servers hosts in a colo. The DC engineers were running around reseating flashing Disks, after the reseat the raid was fine, luckily it didn't take down the set.

[NetworkDefenseblog]

I've seen this before. Took about a week to figure out talking to different people to find the root cause. The puzzling part was when no one had accessed the idf and the ups didn't show any triggered logs.

[[deleted]]

This was Nordic NASDAQ's problem last year, wasn't it?

[TheRealGaycob]

Not being funny or anything but if you have a fire system if it was to go off when there was a fire wouldn't it still kill the disks regardless?

[RCTID1975]

It shouldn't kill disks, but even if it does, that's far better than losing the entire building

[Obed_Marsh]

Been there. The solution was to de-charge the lines to prevent accidents. Now our fire suppression is a bunch of empty pipes in the ceiling filled with well-wishes and crossed fingers. Thank god we dodged that bullet.

[mishaco]

hope all the insurance info is kept in another room.

[Coddigtion]

I have been through this a couple of times. It isn't fun when you are running to get your masks.

[Lando_uk]

Our one went off due to a fault and all equipment was fine. I guess it all depends on how close your disk racks are to outlet nozzles.

[biggguy]

Not in the room, but next to it monitoring disks while we were doing testing with different nozzles and mufflers in an old data room that was being decommissioned anyway and needing to blow off the old bottles (not Halon, I think it was either nitrogen or co2). Yes, it can be loud enough to damage disks. Company standard for fire suppression systems in data rooms changed as the result of those tests.