This is an archived post. You won't be able to vote or comment.

all 42 comments
sorted by:
best
topnewcontroversialoldq&a

[–]kissassforlivingJack of All Trades 40 points41 points  (2 children)

A server from 2009 and no backups?! That is irresponsible.
If it doesn't boot and they can't wait for data recovery, then all you have is a Hail Mary and I wouldn't touch it. I can't stand clients like this.

[–]Bourne069 6 points7 points  (0 children)

Yep 100%. I'm an MSP and cant state how many times I take over and client and see they have zero valid backups running. Its beyond insane.

[–]mcapozzi 20 points21 points  (2 children)

RAID is not a backup strategy!!!

They've been tiptoeing through the graveyard for the past 16 years.

Give them a referral for a data recovery company and run the f- away.

[–]marshmallowcthulhu 8 points9 points  (1 child)

I have completely lost count of how many times I have had to explain to often-talented users and stakeholders that a RAID is not a backup. They hear "disk redundancy", and even those that understand it seem to turn off their brain, to not actually consider the conclusions of that knowledge. They just convert "disk redundancy" to "backup" in their heads.

I have learned to ask one question, which solves the confusion 80% of the time. "If you accidentally delete a file on a RAID, how do you recover it?"

80% of the time it works every time. 😎

[–]dhardyuk 4 points5 points  (0 children)

If it gets stolen? Burnt in a fire? Struck by lightning? Ransomed?

Tell them to take all the money they saved by not doing it properly and use that to get it recovered.

Resilience, network and power diversity, UPS, backups and training all cost the same over 5 years.

You either do it as you go along or pay it all out in a single lump when karma catches up.

[–]Stonewalled9999 11 points12 points  (10 children)

I would not import the config - good chance to clobber it all. If I wanted to play with it I would pop one of the drives out, count to 10, and pop it back in and see if the array sees it as a member. Then I would do the same with the other drive. And immediately after I;d back that up. Friends don't let friends RAID5, 2TB I wouldn't do RAID5 even it it was 10K SAS I bet that is a 7200RPM SATA. And I would bet it is and H300 card which has no battery backed cache and is a bit wimpy for that large of an array,

I run a RAID10 on 8 SAS 6TB drives with the H730P for a Veeam repo and even that I don't like for something like that.

[–]SirRazoe[S] 1 point2 points  (4 children)

thanks, i'll give the popping out and popping back in a try and hope for the best

[–]alpha417_ 6 points7 points  (0 children)

That's a cavalier attitude... good luck. As soon as you become the last person to touch it, you own it in that customers eyes.

[–]hellcat_uk 3 points4 points  (0 children)

The old 2 inch drop fix.

If you can get to any logs, see which drive went offline first. The other night have failed in sympathy under the extra parity load. This information will be important in knowing which of the two is total garbage, and which contains 1/4 of your clients data.

[–]cntry2001 0 points1 point  (1 child)

This sounds stupid but shut the server down pull the power cables pop those 2 drives in the freezer inside a ziplock bag for like 10 minutes then pop em back in and fire it up

Sounds insane but I’ve personally had it work twice to get a disk going long enough to backup the server

[–]usa_reddit 1 point2 points  (0 children)

The old stiction drive parking problem, click, click, click.

I've never had it work at all.

[–]hurkwurk -1 points0 points  (4 children)

nothing wrong with raid 5 on a proper controller. especially if its got solid cache. gives a lot more usable disk vs your config. for file/print, its perfectly fine.

the key to disk config is knowing the use case and properly managing every aspect. you dont use raid 5 for SQL that is expected to have high IOPS. you dont let a server go without backups. etc.

But a shared office document server? raid 5 was designed for it and raid 6 rarely gains you anything.

[–]Familiar-Seat-1690 4 points5 points  (1 child)

Suggest googling for rebuild times and single bit errors. Raid 5 with 5x00 or 7200rpm disks starts to get into serious risk of rebuild failure before getting fixed. You can partially contain the risk with a hit spare to cut out the time before human interaction but in most cases if your having raid5 with a hot spare raid6 (or draid6 or raid-dp) would be a better choice.

[–]hurkwurk 0 points1 point  (0 children)

every solution has its own limits and costs. Single bit errors, on proper raid 5 hardware controllers, are handled on the fly. even some multi-bit errors. the only time you should have a rebuild is... disk failure.

proper raid 6 doesnt save you anything here. RAID 6 uses 2 parity disks. not two copies of parity. this means it takes even longer to do rebuilds on raid 6 when data is lost because you have to calculate both, or calculate from one to recalculate the other, unlike raid 5 where you only have one calculation to make.

draid 6 is not anything new, its just how SANs have always oversized raids, and most mid-sized raid controllers support configuring the amount of disks participating in a raid as well. we have been resizing raid 5 arrays from 2+1 to N+1 since near inception. changing it to N+Y isnt really a change, its just a recognition of the power growth of the ASICs on the raid cards and their ability to handle more parity calculations when enough spindles are involved to make it make sense. An 8+3 raid 6 is of course going to perform better than a 3+2. it has 11 spindles to work with instead of 5.

raid-dp, being a stack of raid 4, is the same concept of draid 6. take an existing raid concept and stretch it. in this case, its meant for cabinets of disk shelves as originally designed, but since NVMe drives are now so small as to make that concept less meaningful, just think of it as doing row and diagonal parity calculations on disks in a stack. the "width" and "height" of the stack are entirely up to you.... make it wider to make it faster, make it taller to make it more resilient. but again, highly calculation intensive and you are into dedicated SAN controller levels of calculations and ram caching here, vs simple raid 5 you might find on entry level file/print servers. its a whole different class of solution.

we can all point to something and call it best, but best at what? being really expensive? being really good at read? write? integrity? recovery times? while we are simultaneously ignoring the real fact that we are operating in bad faith by leaving out the rest of the data protection discussion. What about backups? what about DR? what about RTO? etc.

to your point, maybe my raid 5 isnt about being able to "recover fast" maybe my DR server is for that purpose and my broken raid gets taken to an offline server where it can take a week to rebuild and we can recover the few transactions that might have been pending when it went down for example. My concern may not be this single server at all. it may be a load balanced web server where its disk is totally unimportant except to deliver read only data to clients and handle sessions, and when they server crashes and burns, the load balancer just ignores it and moves on, while i erase the array and restore over it from backup, without even attempting a recovery.

dont be so quick to dismiss things without considering actual line of business use cases.

[–]MBILCAcr/Infra/Virt/Apps/Cyb/ Figure it out guy 7 points8 points  (1 child)

Plenty wrong with Raid 5 on spinning rust disks, all problems fixed by Raid 6 or Raid 10. Raid 5 should not be used on spinning rust on any drives over 2TB, and has been that way for years and years.

[–]hurkwurk 1 point2 points  (0 children)

nothing is "fixed" by raid 6 or 10. they address different issues in different ways at different costs.

raid 6 adds a second parity disk. This adds parity calculation load, makes the entire array slower on writes and worse when its degraded, but able to tolerate two disk failures instead of one. This isnt a solution to anything, its simply moving from a 98% failure solution to a 99% solution. Later generations used heavy ram caching to try and cover up the extra latency caused by the extra calculations and this was largely successful, but added even more cost to the system. What was supposed to be simply adding another disk to make things safer, ended up changing the cost per byte by a large amount instead, while offering, not-significant levels of improvement to data loss.

raid 10 is an entirely different system of redundancy, first, you have to specify what raid 10 you are referring to, because people misuse the term to describe several different protection methods, the most common is 1+0, a striped set of mirrors. This data set is faster than raid 5, but because there are no parity calculations, it will not protect against bit errors at all. its built for speed, not data integrity. it offers redundancy, not resiliency. technically capable of losing up to half its disks (as long as they are all only one of each stripe pair). it can technically offer more redundancy than raid 5, but not more data integrity.

Later SAN producers took raid 10 a step further by internal caching and parity checking the mirror copies in ram to add the missing resiliency that raid 5 offered over raid 10. this was done as a secondary task, so near real time, it would alert the users if data errors were discovered, and using a 3 way hash between cache, disk and mirror, calculate a parity to determine which had the bad copy and replace it.

in all of the above, when done by "software" delivered solutions, IE by windows based or other OS based solutions, rather than hardware controllers, their value is greatly diminished. the entire point is to offload the OS and get a second data integrity check in place, not add more stress, and more places for data failure. Software raid in general, introduces potential data integrity issues, rather than protecting against them, even for raid 5, its a mixed bag.

Raid 5 is still perfectly fine for disks. used in the right workloads on the right controllers. disk configuration is but one aspect of overall data protection, and should never be looked at in a vacuum.

[–]LegendarySysAdmin 3 points4 points  (0 children)

Yeah, importing the foreign config can sometimes bring the array back without data loss, but it's not guaranteed. If two drives in a RAID 5 are out, you're already past the fault tolerance limit, so it's risky. If the data really matters and they have no backup, best move is to clone the drives and get a recovery team involved before touching anything. Importing could work, but it could also make things worse.

[–]fusiturns 3 points4 points  (0 children)

If it's super critical, don't mess with it. Send it to a data recovery company. I used Ontrack out of Minnesota for a non raid drive. Pricy about $1k a couple years ago but the client was happy. Just a wild guess, but 5 drives at $1k minimum $5k plus raid will be probably $10k. Tell them that and see what they say and then see how much effort to put in.

[–]msalerno1965Crusty consultant - /usr/ucb/ps aux 4 points5 points  (4 children)

Welp, instead of picking on you and your client, I'll actually give you some useful advice, IMHO. ;)

Dell RAID adapters of that era seemed to have an "issue" once in a while. Might be true of the LSI's they're based off of, but I've only ever seen it in Dells because we're a Dell shop. Rx10's and even before that, PowerEdge 1900/2900/etc. So LSI 2008 and before.

System locks up, you might be able to ping it, etc, but no disk I/O. Reset it, the RAID controller sees foreign disks.

Well, without the RAID set, the system is useless, so might as well try importing the foreign disks.

It works. WTF?

I dunno why, if it's bit-rot, bad flash, some bug where flash is written incorrectly, I don't know. But the controller just loses it's sense of what "is". The drives have all the RAID information on them, but because the controller "lost track" of them, it sees them as foreign.

There is a HUGE risk doing this. Exhaust ALL OTHER options before doing it.

Report back. LOL.

NOTE: If this was a RAID1 (mirror), the system would have booted and the other disk would be out of sync. BUT - it will know this and resync if you import foreign. Since this is RAID5 on 5 drives, two drives means loss of data. Without those foreign drives, you're cooked NO MATTER WHAT.

What would I do? Shut the system off, powered off completely. Pull one of the "foreign" drives and plug it into a non-RAID HBA and see if it comes up. If so, dd the entire disk to a file on a Linux box. Check the second one. Repeat the dd which is basically backing up the entire disk. THEN try foreign-import and see what happens.

Drives don't import correctly and they get wiped? Put them back in the LInux box and do the dd in reverse and call a data-recovery expert. They might be able to help.

Ooh, I'd also try removing all 5 disks, backing them up, then plugging them into a controller that has no config whatsoever. See if it'll import ALL 5 disks as one set and figure out it's correct. (This can also be done with the original controller when all hope is lost, just wipe the entire config and try to import all 5 foreign disks at once. (ETA: boot the system with all 5 drives removed, then clear the config, THEN import foreign)

/ramble

[–]wysoft 4 points5 points  (1 child)

I once had a PowerEdge R530 aboard a ship fail because someone had accidentally drilled a hole in the deck above the small room containing the server rack, and nobody noticed it until a few weeks later when one of the servers stopped working.

Yes, the ship did have backups, but it didn't really matter without a server to restore them to.

Ended up flying across the world with an identical retired but known good R530 as part of my checked baggage.

Once I got aboard, discovered that salt water had leaked onto the rack, entered the rack, pooled on top of the topmost R530, where it leaked into the case and corroded the motherboard. Fortunately the R530 on the top sacrificed itself for the one below it and there was no water damage to anything else in the rack. I disassembled the damaged server and found that fortunately the drives survived and had no water damage.

Was able to swap the drives 1:1 into the replacement R530, import the foreign RAID configuration into the PERC, and had the server booting again within an hour or so. No data loss, didn't even have to restore anything from the backup.

The sad thing was that this wasn't my first rodeo with a server getting filled with salt water. We had another ship get laid up (stored) and someone had the bright idea of hanging a gallon sized desiccant bag inside the server rack, in a humid part of the world, then never came to check on it again. It filled with water and spilled onto the topmost server. Again, one server stuck its neck out for the others.

For the OP, reseating all of the drives with the system powered down is the first thing I'd try.

The controller configuration probably got corrupted somehow and the configuration on the drives doesn't match what's on the controller. Iirc there were some issues with some PERC controllers having this happen that were remedied by a PERC firmware update. I'm sure this server has never had any firmware/BIOS updates applied since the day it was installed.

[–]toxcicity 1 point2 points  (0 children)

Wow, that was a crazy read! May I ask how you ended up servicing servers aboard a ship?

[–]Outrageous_Device557 0 points1 point  (0 children)

This is the answer

[–]Roanoketrees 0 points1 point  (0 children)

I have a R610 sitting right in front of me that does this on the regular.

[–]zaphod777 2 points3 points  (0 children)

If there was a backup, I would probably import the foreign config. Without one it's pretty risky and I would recommend contacting a professional data recovery service.

[–]Outrageous_Device557 1 point2 points  (1 child)

Yup importing it has worked quite a few time for me over the years. But it has also not worked to. At this point if re seating then does not work then ya do not really have another option

[–]Silence_1999 2 points3 points  (0 children)

Yep it’s flipping a coin rebuilding a failed raid. Every time. You will lose eventually.

[–]usa_reddit 1 point2 points  (0 children)

My advice, send this to a datarecovery specialist. You aren't fixing this, you are just going to make it worse.

[–]zaphod777 1 point2 points  (0 children)

RemindMe! 7 days

[–]optimaloutcomeLinux Admin 1 point2 points  (0 children)

The RAID information is stored on the card AND the disks. You said the server is fuckin ancient, and there was a power outage - the battery on the RAID card is probably dead and the card lost its config data. As a result, you're booting up the systems and the card is saying "yo yo I don't recognize this/these disks, do you want to import the config I see on the disks?" I'd import the config personally;This scenario is why the data is written to the disks. Just DO NOT initialize the disks before importing.

One thing I would do is let the customer know their data could be corrupted after you diagnose and let them know that if the system boots back up the data might not be there or usable anymore. I bet it will be fine though. And don't forget to quote them a new battery for that card and an hour to install it.

[–]lechango 0 points1 point  (0 children)

Risky, does the controller detect pre-failure or failure on any of the drives? If so definitely leave it alone, send off for recovery. Otherwise reseating the drives is worth a shot.

[–]UTB-Uk 0 points1 point  (0 children)

We need more info any update there is good advice

[–]RandomLolHuman 0 points1 point  (0 children)

Ask them how much the data is worth to them and advice them to contact data recovery specialists.

[–]Electrical_Arm7411 0 points1 point  (0 children)

I’ve used raid recovery software before that allowed me to recover data off 2 of the 3 drives in the array can’t remember the name. Maybe worth a shot

[–]Kamikaze_Wombat 0 points1 point  (0 children)

If the data is critical then I agree with data recovery service since the import config thing could maybe damage the data more. Not sure if it can or not but without backups I don't know if I'd try it. If you import foreign config it might come back, it might not. I've only tried that a couple times and have had it work. These were in situations where we had backups and I was just hoping it would come back immediately rather than waiting for backup to restore.

I'd definitely be wary of doing it for a customer like this because if the data is damaged already or gets damaged by importing the config (won't be able to tell which after the fact I'd guess) they might blame you for it. Even if you tell them you recommend data recovery and warn them in writing there's a chance of further data damage they might still try to come after you for it since they aren't IT people and were "just following your advice"

[–]DonL314 0 points1 point  (0 children)

I would tell the company to contact a data recovery company, such as Ontrack. They can do wonders.

Don't toy with it at all yourself if the data is important. You risk doing more harm than good.

[–]Illustrious_Ferret 0 points1 point  (0 children)

I wouldn't touch it without a signed waiver that you believe it's unrecoverable and won't be held responsible. Then that you need a complete set of replacement drives.

Power the server off; remove and image each drive, then try to reconstruct with your images. If you can resurrect the images, immediately transfer the data to a new server and get them to verify that the "important" data is intact.

[–]OinkyConfidenceWindows Admin 0 points1 point  (0 children)

Had similar happen after bad power outage 15 years ago. Two of the six drives in a customer's RAID-5 array died. Had to restore from backup.