I'm going to need some expert help - free beer/coffee to anyone who get's me on my way!
My System I'm running Ubuntu 11.10 with a 3 disk software RAID 5 configuration, with 3TB hard drives, in ext4 format.
What I Did I recently reviewed the health of the disks in the Disk Utility, and saw that two of my drives had bad sectors. Everything was working fine, but I got a little worried. When I looked at the mdadm --detail, I could see that one of the drives was in failure, and the raid was running degraded. However, one of the drives with a few failed sectors was in fact not reporting a failure by mdadm.
I bought a new hard drive, and followed the steps to replace a failed drive in a RAID 5 software configuration. (I marked the drive as failed, removed it, turned off computer, replaced it, partitioned the new drive, and added it back to the RAID). The process nearly completed (I think), but when I reviewed mdstat detail again, things looked far worse than they were before.
What I Have Now
Here is the output of mdstat --detail /dev/mda127
root@mediapc:/home/jason# mdadm --detail /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Sun Mar 10 08:57:16 2013
Raid Level : raid5
Array Size : 5860530176 (5589.04 GiB 6001.18 GB)
Used Dev Size : 2930265088 (2794.52 GiB 3000.59 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Sun Mar 16 06:48:06 2014
State : clean, FAILED
Active Devices : 1
Working Devices : 2
Failed Devices : 1
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 512K
Name : mediapc:127 (local to host mediapc)
UUID : ffbed825:f397afb1:86535cd8:64f8c314
Events : 373600
Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 0 0 1 removed
2 0 0 2 removed
3 8 32 - faulty spare /dev/sdc
4 8 17 - spare /dev/sdb1
My newly added drive (dev/sdb1) was added as a spare. The partition that was reporting some sector issues (in Disk Utility) but was working ok in RAID is now reporting as a faulty spare.
I'm pretty sure all/most of the data is still present, I just have no way to get to it. I'll buy more drives (external) if anyone has an idea on how to get my data back, or how to get my raid running again, if at all possible.
I wanted to try and fix the sectors on the now-faulty spare drive (/dev/sdc) but fsck fails a couple of different ways:
root@mediapc:/home/jason# fsck /dev/sdc
fsck from util-linux 2.19.1
fsck: fsck.linux_raid_member: not found
fsck: Error 2 while executing fsck.linux_raid_member for /dev/sdc
OR:
root@mediapc:/home/jason# fsck /dev/sdc1
fsck from util-linux 2.19.1
e2fsck 1.41.14 (22-Dec-2010)
fsck.ext4: Device or resource busy while trying to open /dev/sdc1
Filesystem mounted or opened exclusively by another program?
I tried added both drives back to the array as they are, but both fail to add:
root@mediapc:/home/jason# mdadm --add /dev/md127 /dev/sdb1
mdadm: Cannot open /dev/sdb1: Device or resource busy
I tried search on way to get fsck to run, but I didn't want to reboot the system until I was sure that was the next correct move.
Really looking for some expert help. I want my data, and I want to get this RAID 5 working - but first and foremost I'd like the data. Thank you in advance. Let me know if there is anything else I can provide.
Regards, Jason
