My RAID Failed

Graham-Elliot Steinhoff 07-17-2014_6554Things were starting to come together for my trip back home. I wrapped up the final revision of the Smelterville book, got a haircut, the van got an oil change and new rear shocks, we went out to say goodbye to Grandsons Graham and Elliot, and I made a lot of progress pulling together a video this morning. I even worked in a well-deserved nap after all that activity.

Then, I looked down at the corner of my computer screen and saw an unusual scary-looking alert message. I think this email to Computer Geek Sons Matt and Adam will tell you more than you ever wanted to know about my travails. Bottom Line: I’m out of business until the Geeks parachute in.

HELP!!!

Computert guts 07-18-2014_6567Got an alert this afternoon that my RAID 1 array had gone critical. That’s my boot drive.

I confirmed that my overnight backup had run and copied a few files I knew had changed over to the Drobo. Then I created an Acronis boot disk.

I loaded up the AMD RAID utility and confirmed that it WAS RAID 1 that was sick. My RAID 5 data drives were fine.

I went to Best Buy and picked up a new drive, just in case.

I told the utility to rebuild the mirror. It would crank awhile, then the computer would reboot.

I pulled what I thought was the drive attached to Port 2, which I thought was the bad drive, then I booted up the computer. It booted fine (much faster than usual, in fact).

There’s no joy in Steinhoffville

I went into the utility and confirmed that the drive in Port 2 was gone.

I put the new drive in, and got an error message telling me to tell it where it should boot from.

I tried Ctrl-F, Esc and F8 to get to a window that would let me do that, but no joy.

I pulled the new drive. Same result. I put the old drive back in. Same result.

I put the Acronis boot disk in the drive. It booted up with a bunch of data dumps and got me to a screen asking if I wanted to recover my computer. Rather than take a chance on screwing something up, I turned off the machine and typed this message.

Matt is getting up a 4 a.m. to ride 130 miles, so I don’t think he’s going to be of much help on Saturday. I’m getting ready to leave town, so I need to get fixed as quickly as possible. Am I overlooking something simple?

Remember the big Amazon button

Buy From Amazon.com to Support Ken SteinhoffClick on the big red Amazon button when you order stuff online. That helps me keep feeding hard drives into my computer and it doesn’t add anything to your cost.

 

Northbound Towboats

Mississipi River BargesThese northbound towboats pushing a string of barges were off Marquette Island south of Cape Girardeau when I shot them sometime around 1964. The white smoke at the top left is from the cement plant.

Click on the photo to make it larger.

Like running in molasses

I’m still recovering from the data drive that went south. The backup restoration has been running for more than 24 hours and has recovered all but about 10,000 files out of 487,776. It’s still churning away, but because the rebuild is taking 99% of the CPU cycles, memory and disk access, everything else is running like that nightmare I have from time time time: the one where the bad guys are chasing me, but I’m running like I’m in a swimming pool full of molasses. I think I’d prefer that to a slow, non-responsive computer.

I hope things will be back to normal by Tuesday morning.

 

One of Those Days

Ellilot Steinhoff with 1st Birthday cake 02-08-2014Saturday was a dual birthday party for Grandson Elliot, who just turned one, and Grandson Graham, who will be three on Valentine’s Day. Son Matt came over Sunday to fix some computer issues I was having. This is how I looked when he told me that he had wiped out my data drives.

My mood will be better in 24 hours

Ellilot Steinhoff with 1st Birthday cake 02-08-2014Fortunately, I had a full backup that finished at 4:56 Sunday morning. After several false starts, I called Son Adam, Dad of Elliot and Graham, who gave me some hints on how to restore the data. Nineteen to 24 hours, 487,776 files and 2.34 Tb from now, I should look like Elliot at his happy stage. Had the local backups not worked, I would have contacted Backblaze and had them send me a 3 Tb portable drive with all my files on it. Downloading that many files would have taken months.

Remember the Red Button

Buy From Amazon.com to Support Ken SteinhoffSorry for the short post. Most of everything I’ve been working on for about the past five years is flying back an forth between the backup drive and the new ones. Thanks to all you folks who do your Amazon shopping by clicking that big red CLICK HERE button at the top left of the page (or this one). The small percentage I make off your purchases and the help from my kids are what keep this thing running. (When they aren’t breaking stuff while fixing it.)

Backup, Backup, Backup

I’ve had a very frustrating (and expensive) last two days. I’m a pessimist who believes that Murphy wasn’t an only child. I usually have not only a Plan B, but Plans C through E. In fact, a pessimist is someone who is actually DISAPPOINTED when Plan A works.

Optimists, on the other hand, don’t have Plan Bs because they are SURE than Plan A is going to be successful beyond anyone’s dreams.

How do you fight Murphy?

I have about 50% of my data on a pair of external 2-terabyte drives. The drives are mirrored, so the information is duplicated on both drives at the same time. If one fails, then you slap in a replacement and the mirror rebuilds itself.  We used this technology on equipment in our telephone switchroom, and it saved our voicemail system and what we called the “cash register,” the equipment that supported our circulation and classified advertising call centers. There’s no more sinking feeling than seeing the alarm, “Drive Fail>’

On the other hand, there’s a great feeling of satisfaction when you slide in the spare and watch your world – and your job prospects – become infinitely brighter.

Backblaze puts out the fire in your tummy

To be even safer, I back up the mirrored drives to a second external drive, and I also use on offsite, “cloud” backup system called Backblaze. You can’t beat the deal. It costs five bucks a month and you can back up unlimited amounts of data. (It was an even better deal for me: Son Adam prepaid a year of the service as a Father’s Day gift.)  The advantage to a cloud backup is that it’s not in your house where it can get stolen, flooded or burn up.

I wasn’t their normal customer. I have so much data that it took about four months to upload it all. One it’s there, though, it constantly monitors the files on my computer and sends changes to the cloud almost immediately.

I just signed up to become an affiliate, so if you click on the Backblaze logo above, or this link, I’ll get credit if you sign up for the service.

If you are an optimist, this is a good Plan B. If you are a pessimist, this will probably slot in at about Plan C or D. Five bucks a month is less than some folks spend in Starbucks a day and it’ll let you sleep a lot better.

So, what happened?

Last month, my RAID drives gave me an alarm that one of them had taken a dirt nap. These drives have names. The bad guy was the primary, the good guy was the secondary.

I ordered a replacement drive under warranty. It arrived in a couple of days. I popped it into the slot and watched with satisfaction as the primary synched up to the secondary in about six hours. All was good for about a month. Then, two days ago, the drive in that slot failed again. The vendor had sent me two drives by mistake, so I pulled out the replacement and replaced it with the second drive.

Just before I turned the drive back on, I decided to do a backup on my external drive G. I kicked it off just before going to bed and it finished just about breakfast time. That means the data exists on the secondary, on Drive G and in the cloud with Backblaze.

Life just got uglier

I pushed the new drive in, powered up the enclosure and got an error message that the drive had to be formatted. Format is a scary command. If you screw it up, you’ll wipe out all the data on the drive. To be on the safe side, I removed the good secondary drive, then formatted the primary. I’d never had to do that before, but…

I put everything back together, turned on the power, then watched, first with satisfaction, then with horror, that the mirror was being rebuilt. The little arrows were going the wrong way. The little arrows SHOULD have been pointing FROM the secondary to the primary. Instead, they were going the other way, meaning that the blank primary drive was overwriting all my data.

The pessimist in me was satisfied

This was a bad thing, but, at least I had the satisfaction of knowing that I had two other sources of the data, both fresh. I logged into my Backblaze account and started a download of the data. It’s a lot faster to restore from a local drive than from the cloud. (If I had REALLY been in a hurry, Backblaze would have sold me an external drive with my data on it.) Downloading everything on that drive was going to take about 50 hours, even with a fast Internet connection.

Here’s where I made a minor error. Backblaze has everything that was on the drive. When I did my backup to local Drive G, I elected not to copy over a couple of directories of nice-to-have-but-not-essential files. I would have been better off to have requested a Backblaze download of only those files instead of EVERYTHING. It would have saved a lot of download bandwidth and time. Still, this is a nice practice run and will give me a good idea of how good the service is.

THEN what happened?

I decided that the problem was probably in the piece of equipment that holds the drives, not the drives. I pulled out the primary drive, booted up on only the secondary, and copied all my data from the local external drive to the secondary. It worked fine. I decided NOT to try to rebuild the mirror.

I called Son Matt, who has been trying to convince me that I should buy a magic Drobo S Beyond Raid 5-Bay USB 3.0/FireWire 800/eSATA/SATA 6GB/S Storage Array with Drobo PC Backup by Drobo  because he had good luck with them both at home and at work. The magic part is that you can put a mixture of drive sizes in it to use old drives or you can upgrade them if you need more storage. Wife Lila is out of town on a cruise ship in Alaska, safely out of cellular range, so I felt safe in ordering the Drobo.

The bad news is that it cost $553.14 (without the drives). (That’s one of the reasons you should click on my Amazon link at the top left of the page. It helps pay for these kind of glitches.)

The worse news is that I clicked on the item to create the links on this page, and saw that the price had DROPPED in the few hours since I had placed my order. A very nice woman said they don’t normally do price matching, but they’d make a one-time exception for me and refund the difference between $553.14 and the new price of $518.49.