Backup, Backup, Backup

I’ve had a very frustrating (and expensive) last two days. I’m a pessimist who believes that Murphy wasn’t an only child. I usually have not only a Plan B, but Plans C through E. In fact, a pessimist is someone who is actually DISAPPOINTED when Plan A works.

Optimists, on the other hand, don’t have Plan Bs because they are SURE than Plan A is going to be successful beyond anyone’s dreams.

How do you fight Murphy?

I have about 50% of my data on a pair of external 2-terabyte drives. The drives are mirrored, so the information is duplicated on both drives at the same time. If one fails, then you slap in a replacement and the mirror rebuilds itself.  We used this technology on equipment in our telephone switchroom, and it saved our voicemail system and what we called the “cash register,” the equipment that supported our circulation and classified advertising call centers. There’s no more sinking feeling than seeing the alarm, “Drive Fail>’

On the other hand, there’s a great feeling of satisfaction when you slide in the spare and watch your world – and your job prospects – become infinitely brighter.

Backblaze puts out the fire in your tummy

To be even safer, I back up the mirrored drives to a second external drive, and I also use on offsite, “cloud” backup system called Backblaze. You can’t beat the deal. It costs five bucks a month and you can back up unlimited amounts of data. (It was an even better deal for me: Son Adam prepaid a year of the service as a Father’s Day gift.)  The advantage to a cloud backup is that it’s not in your house where it can get stolen, flooded or burn up.

I wasn’t their normal customer. I have so much data that it took about four months to upload it all. One it’s there, though, it constantly monitors the files on my computer and sends changes to the cloud almost immediately.

I just signed up to become an affiliate, so if you click on the Backblaze logo above, or this link, I’ll get credit if you sign up for the service.

If you are an optimist, this is a good Plan B. If you are a pessimist, this will probably slot in at about Plan C or D. Five bucks a month is less than some folks spend in Starbucks a day and it’ll let you sleep a lot better.

So, what happened?

Last month, my RAID drives gave me an alarm that one of them had taken a dirt nap. These drives have names. The bad guy was the primary, the good guy was the secondary.

I ordered a replacement drive under warranty. It arrived in a couple of days. I popped it into the slot and watched with satisfaction as the primary synched up to the secondary in about six hours. All was good for about a month. Then, two days ago, the drive in that slot failed again. The vendor had sent me two drives by mistake, so I pulled out the replacement and replaced it with the second drive.

Just before I turned the drive back on, I decided to do a backup on my external drive G. I kicked it off just before going to bed and it finished just about breakfast time. That means the data exists on the secondary, on Drive G and in the cloud with Backblaze.

Life just got uglier

I pushed the new drive in, powered up the enclosure and got an error message that the drive had to be formatted. Format is a scary command. If you screw it up, you’ll wipe out all the data on the drive. To be on the safe side, I removed the good secondary drive, then formatted the primary. I’d never had to do that before, but…

I put everything back together, turned on the power, then watched, first with satisfaction, then with horror, that the mirror was being rebuilt. The little arrows were going the wrong way. The little arrows SHOULD have been pointing FROM the secondary to the primary. Instead, they were going the other way, meaning that the blank primary drive was overwriting all my data.

The pessimist in me was satisfied

This was a bad thing, but, at least I had the satisfaction of knowing that I had two other sources of the data, both fresh. I logged into my Backblaze account and started a download of the data. It’s a lot faster to restore from a local drive than from the cloud. (If I had REALLY been in a hurry, Backblaze would have sold me an external drive with my data on it.) Downloading everything on that drive was going to take about 50 hours, even with a fast Internet connection.

Here’s where I made a minor error. Backblaze has everything that was on the drive. When I did my backup to local Drive G, I elected not to copy over a couple of directories of nice-to-have-but-not-essential files. I would have been better off to have requested a Backblaze download of only those files instead of EVERYTHING. It would have saved a lot of download bandwidth and time. Still, this is a nice practice run and will give me a good idea of how good the service is.

THEN what happened?

I decided that the problem was probably in the piece of equipment that holds the drives, not the drives. I pulled out the primary drive, booted up on only the secondary, and copied all my data from the local external drive to the secondary. It worked fine. I decided NOT to try to rebuild the mirror.

I called Son Matt, who has been trying to convince me that I should buy a magic Drobo S Beyond Raid 5-Bay USB 3.0/FireWire 800/eSATA/SATA 6GB/S Storage Array with Drobo PC Backup by Drobo  because he had good luck with them both at home and at work. The magic part is that you can put a mixture of drive sizes in it to use old drives or you can upgrade them if you need more storage. Wife Lila is out of town on a cruise ship in Alaska, safely out of cellular range, so I felt safe in ordering the Drobo.

The bad news is that it cost $553.14 (without the drives). (That’s one of the reasons you should click on my Amazon link at the top left of the page. It helps pay for these kind of glitches.)

The worse news is that I clicked on the item to create the links on this page, and saw that the price had DROPPED in the few hours since I had placed my order. A very nice woman said they don’t normally do price matching, but they’d make a one-time exception for me and refund the difference between $553.14 and the new price of $518.49.

 

 

 

12 Replies to “Backup, Backup, Backup”

  1. You have the intelligence and experience to work through this kind of nightmare. Even though Don has excellent computer skills, whenever I get into this kind of situation, I pick up my equipment and run posthaste to my nearby computer repair store. They have never failed me and it is incredible the number of self-inflicted wounds I make to my computer. I am glad for you that a solution seems on the way.

  2. Ken that sounds like a lot of GEEK, but I don’t have the massive file system that you have. It was a very interesting story though.

    1. I’ll boil it down: even if you don’t have a LOT of data, how much are you willing to lose of what you have? That’s the reason I like the Backblaze cloud idea: I don’t have to remember to back it up; I don’t have to buy additional hardware (even though I’m paranoid enough to do it); and it’s not sitting next to the original data where a local catastrophe could wipe it out.

      Back in the days when we had floppy drives that were sloooow, the rule was, “Save what you’re working on every time that it takes less time to save than to recreate.”

  3. Ken, a clear description of disaster for the non-techs among us – topped with great advice…interspersed with intriguing photos. a loyal fan, Mary

  4. Oh my. That’s a nightmare. And it scares me because as an optimist I have only a Plan B for Kiss and Tell’s manuscript. Every 10 days I back my work up to a hard drive.
    Well, good thing Lila was out of phone reach so she couldn’t veto your extravagance. I know you’re thrilled to have discovered your pessimism was not misplaced in this case.

    1. Think about that a minute: how much writing do you do, how many revisions do you make and how much email do you exchange with your editor in that 10 days?

      How would you like to recreate all that work on Day Nine if you had a crash before your scheduled backup?

      And, is it really every 10 days? Do you sometimes realize that’s it’s been 11 or 15 or 30?

      The fact that you work on a laptop that gets carried around and you don’t have mirrored drives ups the odds of a catastrophic failure.

      Hurricanes and drive crashes: it’s not a matter of IF….

  5. Like both of us, there are a lot of people that have downloaded pictures from their cameras to their hard drives. Unlike you (and now me), most do not have their precious photos protected by having them backed up somewhere else.
    Despite my best practices, a worm virus infected my desktop computer and in the process of getting the whole thing fixed, I mistakenly formated (low level; absolutely unrecoverable) the wrong hard drive. There went seven years worth of marching band photos as well as nearly every digital photo I had ever taken. Fortunately, from multiple sources, I have been able to recover all but a few. I was also very fortunate because my Model Railroad files WERE backed up in two external locations.
    I think I am going to take advantage of your link to BACKBLAZE and be smarter still….

    1. I’m not going to say Backblaze is the answer to everything. Their business model seems to be pretty good, but I’d be afraid to keep ALL my valuable stuff with anybody in the cloud. Maybe they go out of business or decide to triple their prices or charge guys like me who have a lot of data what the storage space is really worth.

      Still, if was really comforting to know that I had a third place to go to for the data.

      Once all of the data has finished downloading in another 21 hours, I’ll know how enthusiastic my endorsement is going to be. I can’t complain about the simplicity of the backup process. The restore is going to be the key. I don’t care how many backups you’ve got, if you’ve never made sure that you could restore from those backups, you’re just “salting the elephant.”

  6. Ken, Even I, as a Data Architect at work, often lack the discipline at home to ensure my personal files are backed up. A hard drive failure for me recently on my personal desktop ended up OK only after spending dollars to recover files. I was lucky in that case, but maybe not in the future. I can’t tell you how many times large companies spend a fortune on having a backup procedure in place, but for some reason, their restores do not work…so I would add, testing the restores from time to time right after you make a backup just for that added security.

  7. Ken,
    Used your link and signed up. was getting worried about this topic and basically doing nothing. So this was good. have 2.5 terrabytes of web design crap, so now I can sleep at night. Thanks
    Bob

    1. With that much data, comparable to mine, be prepared to live with the upload for a long time. It starts with the smallest files first, then moves to the big ones. I turned the setting up to the fastest upload speed and never noticed it bogging my Internet speed down.

      Depending on what kinds of files you work with, there are some exclusion tweaks you can do. For example, I noticed that it was uploading all the Photoshop cache files that were buried about a dozen directory levels deep. If I ever had to do a restore, those would be rebuilt on the fly, so I excluded them. I also didn’t see any need to back up movies and shows my Tivo had downloaded to my computer.

Leave a Reply

Your email address will not be published. Required fields are marked *