Recovering from Two Failed Drives in a Six-Drive Raid-5 Array

Over the weekend two drives failed on one of our development machines and I lost a lot of work. The machine has six 300gb drives in a raid 5 array. Our system administrator tells me that if only one drive failed it would be possible to recover. We've tried putting the drives in the freezer for a few hours with no luck. It sounds like sending the array off to a recovery specialist would be very costly and hit or miss. Any ideas?

From serverfault hypoxide

• Restore from backup...

  Matt Simmons : Yep. One of the reasons that RAID-5 isn't encouraged anymore. 2-parity raid schemes are more reliable, and less likely to fail during recovery.
  From Wayne Sheppard

• If you've lost two drives then you've lost your data. The only recourse is to restore from backup if one exists or, as you stated, send the drives off to a data recovery company to try and recover the data.

  hypoxide : What an unfortunate twist of fate to lose two drives in one weekend. The server is designed to be an exact copy of the on-site server, though unfortunately this one is sans backup.

  joeqwerty : In that case then why not replace the failed drives and sync the content from the on-site server?

  John Gardeniers : And when you replace the drives give some serious consideration to adding a hot spare, or creating a RAID 6 array instead.

  Zoredache : @hypoxide, given the size of drives these days loosing two drives is becoming far more likely. See Why RAID 5 stops working in 2009 - (http://blogs.zdnet.com/storage/?p=162)

  hypoxide : @joeqwerty, fortunately the on-site server does have a recent version of the database from which I'm able to restore. However, over the last month I've been developing on our local mirror and that was the only place my work resided. It looks like I'll just have to rewrite the SQL. At least I'll have an opportunity to refactor it.

  Wayne Sheppard : Some type of distributed source control management tool might be useful as well.

  Doug Luxem : This isn't the only recourse. As mentioned in the question, recovery specialists can rebuild RAID from multiple failed drives. My guess is that you are looking well over $3000 for the rebuild though.

  hypoxide : @DLux: That's what I'm afraid of. We do have a contract with a government-grade data recovery specialist and we've dealt with them in the past, but they are extremely costly and don't promise anything. It's likely less expensive for me to do a rewrite than to send the drives in.

  hypoxide : @Wayne Sheppard: importing the latest schema into team foundation server presently.
  From joeqwerty

• Go to the bank and get your backup tapes because it's gonna be a long night. Also, ask for RAID6 with a hot spare next time.

  From Wesley 'Nonapeptide'

• If the drives have not physically failed and were just kicked out of the RAID set you may be able to recover the data. I have had luck recovering data in this situation by using a piece of software called R-Studio (r-studio.com). If the RAID controller in question has a pass-through mode turn this on and let R-Studio scan the drives. It will attempt to rebuild the RAID set and let you restore any non-corrupt data to another drive.

  hypoxide : Unfortunately the drives failed physically, and make a horrible clicking noise when attempting to boot.
  From heydmj

• This has happened to me twice, now I use RAID-6. This is common when after one drive fails, you attempt a restore and another one fails. Your best bet is to attempt a recovery of a bad drive to a blank one of equal or greater size then force it into the array. To restore to a blank drive just do a 'dd if=/dev/baddrive of=/dev/gooddrive bs=1024k conv=noerror,sync'. If there's a bad spot on the drive it will be replaced with zeros, but at least you get 99.99% of your data back.

  From $pageUsers[$post.author].name