Brad’s first blog contest — backup horror stories

And we’re done. I’ve received some excel­lent entries in the Backup Hor­ror Story con­test. Give ‘em a read below, and feel free to add your com­ments too. And of course, feel free to share your hor­ror story too, but sadly, this con­test is closed.

Wel­come to the first (of hope­fully) many con­test I’d like to run on my blog. I’ve been doing this tech-writ­ing thing for a while now, but I’ve always been look­ing for ways to more closely engage with you. I think I may have found it with this style of con­test — I get you to write for me. It’s ok, I have prizes 🙂

I’ll get to the details in a moment, but first let me frame the scen­ario: Backup Hor­ror Stor­ies.

We’ve all been there. We’ve all, at one time or anoth­er, lost some import­ant piece data; your digit­al photo col­lec­tion or music col­lec­tion. Per­haps you’re a writer and all your ‘in pro­gress’ manu­scripts are now toast. You’ve lost data. No extern­al copy or backup avail­able. Poof! Done!!

That’s the scen­ario, now the details:

The Prizes and Spon­sor
Click­free — a cool Cana­dian com­pany that spe­cial­izes in no-brain­er backup solu­tions is spon­sor­ing this con­test and has giv­en me a few Click­free Trans­former SEs for this con­test. Since Click­free is all about simple backups, the theme of the con­test kinda sug­ges­ted itself 🙂

In the past, I’d reviewed Clickfree’s C2 Port­able Backup drive — a sol­id unit. The Trans­formerSE we’re giv­ing away in this con­test uses sim­il­ar tech­no­logy, only you provide the USB drive. Here’s the offi­cial com­pany line on the Trans­fomerSE:

The Click­free Trans­former SE (Spe­cial Edi­tion) turns any USB hard drive, iPod, or iPhone into a simple auto­mat­ic backup solu­tion for your com­puter. Just con­nect the Trans­former SE to your com­puter, then con­nect the USB hard drive, or iPod/iPhone via USB into the Trans­former SE. Backup will start auto­mat­ic­ally onto the avail­able free space of the con­nec­ted product, wheth­er it is a 3rd party hard drive, or an iPod/iPhone.

I will be doing a full review of the Click­free Trans­formerSE very soon, but don’t let that stop you from enter­ing the con­test.

To enter:
Take your worst / best backup hor­ror story and write-up a com­ment to this page that describes a data loss hor­ror story that was aver­ted or would have been pre­ven­ted if you had a trusty recent backup. That simple.

Import­ant: If you’ve not com­men­ted here before, your com­ment may be held in mod­er­a­tion until I can author­ize it. No wor­ries, I do this daily.

The Rules:
I’m keep­ing this fun, so the rules are simple.

1) It’s a blog com­ment con­test — tell me your story in a com­ment to this page using the form below. Any­one can enter. Only com­ments entered into the com­ment form below on this page will be eli­gible.
2) After that you post a com­ment, let me know through a private email noti­fic­a­tion to me (via this in-blog con­tact form). It’ll let me know you’ve entered and be sure to provide a val­id email address for fol­low-up should your entry be selec­ted. No, I won’t sell or spam you..the email address is to be used ONLY for this con­test. After the con­test, all email entries will be deleted.
3) Top 3 com­ments will be selec­ted for a prizes. I’m not sure what cri­ter­ia I’ll use to judge yet. Maybe the fun­ni­est, most dra­mat­ic, most poten­tial for loss-of-life, I don’t know. Maybe the most sup­port from oth­er com­ment­ors (get your friends to help out!). But there will be three, and I’ll write about them in a fol­low-up post.
4) Ran­dom draw for a few more prizes. It’ll be ran­dom.
5) Win­ners noti­fied with­in a week, deliv­ery with­in a month via Canada Post.
6) The con­test starts now (March 1, 2010) and runs until Mid­night, March 31, 2010. Timestamp of the blog and cor­res­pond­ing email to me will determ­ine entry date and time.

Bonus Prize: Every­one Wins
Ok, now this is also very cool. For the month of March, the fine folk at Click­free have also author­ized a dis­count code for orders on their site. Place any order, use this code ( Grier10 ) and they knock 15% off the price of your order.

Related Posts with Thumbnails

17 Replies to “Brad’s first blog contest — backup horror stories”

  1. We do inde­pend­ent game devel­op­ment at work and much of my time works remotely from one anoth­er, we have been both bit­ten and saved by our backup repos­it­or­ies both phys­ic­al and soft­ware.

    One story in par­tic­u­lar involves a pro­gram­mer who felt that once he had his assigned tasks felt that he did not need to com­mu­nic­ate with the team. He would work on his tasks and then check in his files in to SVN. Unfor­tu­nately this par­ti­ular time he was work­ing on a big­ger task that took him a week to com­plete, dur­ing a time when we were mak­ing massive changes to the game. As such when this pro­gram­mer was fin­ished his task, he checked his changes into SVN as usu­al (since they worked fine on his end on an week old ver­sion of the game) and nev­er noti­fied any­one.

    Unfor­tu­nately for the rest of us, we were also work­ing on massive changes to our game and pulling all-night­ers to do it. We updated our loc­al SVN repos and tried to work with the new changes that we were all mak­ing (plus unknow­ingly the changes this oth­er guy made)… only the game ended up crash­ing. It worked fine before this latest update and no one was sup­posed to have made any changes that would cause this prob­lem, and yet, here it was, the game was crash­ing. Franticly we looked at all the changes “we” had made for the prob­lem (remem­ber we did not know this guy had checked any­thing in) and argue­ments rose over who was at fault of this issue (oddly no one fingered the par­tic­u­lar pro­gram­mer in ques­tion since we didn’t know he had com­mit­ted any­thing, plus it was 4am and no one was think­ing straight).

    Being an artist, dur­ing one of the more heated pro­gram­mer debates I took a closer look at the SVN logs and dis­covered that indeed this guy had stealth­ily checked in some­thing between our changes. Luck­ily for us, when you use a ver­sion repos­it­ory sys­tem (soft­ware or hard­ware) you can roll back your changes to a pre­vi­ously uploaded state that’s stored on the device (since all the data gets saved for each change that is made). I gave this a try loc­ally and quickly dis­covered what the prob­lem was and relayed it to the rest of the team. Crisis aver­ted thanks to our backup sys­tem! We ended up perma­mently rolling back the changes this guy had made on the serv­ers and had a pretty healthy dis­cus­sion with him the next day (it was 4am at the time and no one had the effort to put in a nasty email).

    That’s my example of how a backup sys­tem saved our butts and I think a pretty val­id story that sig­ni­fies why people should invest in a backup solu­tion that is more than just an extern­al HDD but can also track changes you have made too.

  2. I got a frantic call from a grad stu­dent once, say­ing that someone had broken in and stolen his com­puter with all his thes­is data and his 34 fin­ished draft thes­is — two years of data col­lec­tion research and writ­ing gone!
    I said, “Didn’t I tell you to back everything up every week at least?”
    And he said, “I did, but they stole my extern­al drive and my back up CDs as well! The cleaned me out! What am I going to do? I can’t face start­ing over! I’ll have to drop out!”
    “Didn’t I tell you to send a copy of your thes­is to you moth­er every time you fin­ished a chapter?”
    “You did!”
    “And did you?”
    “I did! I for­got all about that! I did that just Wed­nes­day! It’s at my Mom’s!”
    And all was well
    Okay, so the mor­al is, the prin­ciple of off­s­ite backup might be a little too abstract for some users, but every­one gets send­ing a copy to Mom.

  3. My own story is a little dif­fer­ent. I backed up everything reli­giously, but about six months away from com­plet­ing my thes­is, it finally occurred to me that I had a stack of flop­pies (and my broth­er and moth­er had stacks of flop­pies at their houses) that could only be read by an Osborne com­puter and that I was the only per­son I knew who still used an Osborne com­puter.… so I went out and bought a backup Osborne. And sure enough, my com­puter died a month or so later. But I thought, no prob­lem, I have my back up, and it sur­vived long enough for me to upload everything to the UofA’s MTS sys­tem. There! Backuped on a main­frame, what could be more secure than that? So it didn’t both­er me when I took my Osborne in for repair and the guy at the counter at the repair place called every­one in from the back­room to look at an actu­al Osborne and they all laughed uproarously at the thought of try­ing to find parts are hav­ing a clue how to fix one.…. I was covered. So I bought an IBM and worked for a couple of years on it know­ing that my data was backed up at the UofA some­where and not wor­ry­ing about it until years later I went to do a fol­low up study and asked for my data and dis­covered that not only is the Uni­ver­sity of Bra­sil­ia the last place on earth still using MTS, but the phys­ic­al com­puter I’d used all those years has been torn down and replaced with a couple of Mac serv­ers.…

    For­tu­nately, I still have everything in hard copy because I am, you know, anal, but that’s still a lot of retyp­ing if I ever get around to using that data again.…

  4. @Runte: yeah, that’s one flaw with backup scen­ari­os. You have to keep the backup on *cur­rent* media.

    A few years ago I backed up my photo col­lec­tion on flop­pies. Then it grew too big for that, so I used CD / DVDs — and don’t have a floppy read­er in the house.

    Some day I expect that I’ll not use CD / DVDs either, so those backups will be worth­less.

    Cur­rently, I use a remov­able 500GB hard drive as my backup device. I have a few of them and swap them between a secure off­s­ite stor­age place (Not moms 🙂 ) Work and home.

    Online backup is also an option these days, though some folk get queasy at stor­ing their data in the Cloud.

  5. I have many backup and data loss stor­ies. Here’s one, to enter the con­test, Brad.

    At a pre­vi­ous employ­er we backed everything up on tape.

    So, one day I find my web logs have become cor­rup­ted.

    I check the serv­er folder and find old logs have been purged.

    I put in a help tick­et to request the tape. Unfor­tu­nately, it’ll take three weeks for the tapes to be retrieved. Ah well, that’s ok. My reports are not a 911 and I’m just happy my data is safe.

    Six (6) weeks later the tapes arrive and reveal my web logs can­not be restored from the backups. Data gone.

    My backup les­son is to always test restore a backup after it’s first been cre­ated. Also do test restores peri­od­ic­ally, because you can reach a fail point even after an ini­tial test.

  6. Hey Johnn,
    Pain­ful, yet good les­sons to learn. Lucky for me I’ve nev­er had a bad backup situ­ation, but yes, I do test them pri­or to com­mit­ting to a sys­tem.

  7. Anoth­er les­son I learned on my per­son­al machine is to always use cus­tom setup dur­ing pro­gram install­a­tion. This often lets you choose your data dir­ect­ory.

    For any­one who doesn’t mir­ror their whole drive as part of their backup pro­cess, they’ll want to ensure their data is in with the oth­er folders they have flagged for sched­uled backups.

  8. Agreed! Cus­tom setup is essen­tial for many reas­ons. In my case I use one phys­ic­al device to keep my OS on, and anoth­er for my pro­gram files, install­a­tion data, etc.

    Makes rein­stalls a little easi­er too.

  9. I work for a large man­u­fac­tur­ing cor­por­a­tion and for years we’ve been using tap backup. So long that the backup serv­er is run­ning Sun­OS 5.8 and Backup Exec 3.4. to a Quantum backup robot the size of two fridges. Appar­ently they can’t be upgraded cur­rently because of the VAX machines still on the net­work that won’t sup­port any­thing else. The backup pro­cess sort of fell on me when I star­ted because well.…I was the new kid in the dept. Being out­dated and all, I just fol­lowed the instruc­tions in the tattered old instruc­tions someone had typed out by some oth­er suck­er that couldn’t remem­ber the long list of steps to get everything done.

    Things seemed to be work­ing fine and I nev­er ques­tioned the backup pro­cess, because I didn’t know enough about what I was actu­ally doing to be able to question.…things. Recently though, a num­ber of things star­ted to peak my curi­os­ity of what I was actu­ally doing. When fin­ance kicked back the latest request for new tapes I star­ted to ask around. Why had we always thrown new tapes into the machine, pulled old tapes out and shipped them to Cal­gary to be stored by Iron Moun­tain? After all these years of doing this, we must’ve had HUNDREDS of tapes in the lib­rary. I nev­er thought about it but there wasn’t any sort of rota­tion for the tapes. This was the start of the prob­lems.

    Next, we real­ized that no one had ever attemp­ted to do a res­tor­a­tion of the data. Upon fur­ther invest­ig­a­tion we dis­covered that it wasn’t just a mat­ter of people not hav­ing attemp­ted it but that we couldn’t actu­ally do a test restore on the old sys­tems without affect­ing the pro­duc­tion sys­tems. Had we wasted hun­dreds of thou­sands of dol­lars in tapes in a vault that could be worth­less?

    And then it happened. One morn­ing one of the man­u­fac­tur­ing man­agers called me frantic­ally because he’d noticed that the piece of junk old mon­it­or we had for the serv­er was on…and say­ing that it couldn’t boot because of drive cor­rup­tion. And of course, I was expec­ted to sup­port the sys­tem to get it oper­a­tion­al again.

    What the crap was I sup­posed to do? I’d played with Linux a bunch and that but I sure as hell didn’t want to test my know­ledge on a sys­tem that old that had been back­ing up data for a mil­lion dol­lar a day oper­a­tion. Nat­ur­ally, I took my sweet ass time get­ting up to the serv­er room. Try­ing to think of what my options were.

    By the time I got to the machine the machine had gathered a small audi­ence. Anxious to know wheth­er or not the junk­ard sys­tem would come back up. It’s always funny when people don’t think they need to spend money on a backup sys­tem until it’s crit­ic­al. I mean, we weren’t in need of the backups them­selves but the real­iz­a­tion that the sys­tem obvi­ously wasn’t adequate star­ted to sink in very quickly.

    My first step was clear, sit down and read the mes­sage again. Obvi­ously I had to con­firm that the “user” was read­ing it right. Wait, what’s this? It says “Hit enter to reboot the sys­tem”? No one had thought to do it, yet.

    Clos­ing my eyes as my index depressed the Enter but­ton caused me to take a deep breath.

    I was pretty sure that no one in the room would ever trust my judge­ment ever again. I’d taken the noob help­desk approach. I hadn’t even come with a note­book or cds as tools to resolve the issue. Nor had I said any­thing “techy” about pos­sible causes, res­ol­u­tions or con­cerns.

    Slowly, I opened my eyes to the Sun­OS logo. A sigh of relief filled the air con­di­tioned room. The sys­tem was com­ing up prop­erly.

    It’s been a couple month since the sys­tem had the prob­lem but it hasn’t mys­ter­i­ously rebooted in that time. We’ve star­ted to recall old tapes from the very begin­ning of the backup pro­cess.

    Now that we’re no longer order­ing new tapes it’s added a hefty amount towards the backup sys­tem that they’ve been appar­ently plan­ning to imple­ment and just needed budget for. It isn’t much in the grand scheme of the pro­ject but the les­sons learned added invalu­able worth to it all.

    Unfor­tu­nately until such a time as the new sys­tem is in place I still have to go through the motions of using the slow and old robot…

  10. @foomanizer: Wow, amaz­ing that an org that runs a $1m/day oper­a­tion would leave backup to antiques like that.

    Very cool that they appear to have seen the light.

Comments are closed.