Very cool that they appear to have seen the light.

Things seemed to be working fine and I never questioned the backup process, because I didn’t know enough about what I was actually doing to be able to question.…things. Recently though, a number of things started to peak my curiosity of what I was actually doing. When finance kicked back the latest request for new tapes I started to ask around. Why had we always thrown new tapes into the machine, pulled old tapes out and shipped them to Calgary to be stored by Iron Mountain? After all these years of doing this, we must’ve had HUNDREDS of tapes in the library. I never thought about it but there wasn’t any sort of rotation for the tapes. This was the start of the problems.

Next, we realized that no one had ever attempted to do a restoration of the data. Upon further investigation we discovered that it wasn’t just a matter of people not having attempted it but that we couldn’t actually do a test restore on the old systems without affecting the production systems. Had we wasted hundreds of thousands of dollars in tapes in a vault that could be worthless?

And then it happened. One morning one of the manufacturing managers called me frantically because he’d noticed that the piece of junk old monitor we had for the server was on…and saying that it couldn’t boot because of drive corruption. And of course, I was expected to support the system to get it operational again.

What the crap was I supposed to do? I’d played with Linux a bunch and that but I sure as hell didn’t want to test my knowledge on a system that old that had been backing up data for a million dollar a day operation. Naturally, I took my sweet ass time getting up to the server room. Trying to think of what my options were.

By the time I got to the machine the machine had gathered a small audience. Anxious to know whether or not the junkard system would come back up. It’s always funny when people don’t think they need to spend money on a backup system until it’s critical. I mean, we weren’t in need of the backups themselves but the realization that the system obviously wasn’t adequate started to sink in very quickly.

My first step was clear, sit down and read the message again. Obviously I had to confirm that the “user” was reading it right. Wait, what’s this? It says “Hit enter to reboot the system”? No one had thought to do it, yet.

Closing my eyes as my index depressed the Enter button caused me to take a deep breath.

I was pretty sure that no one in the room would ever trust my judgement ever again. I’d taken the noob helpdesk approach. I hadn’t even come with a notebook or cds as tools to resolve the issue. Nor had I said anything “techy” about possible causes, resolutions or concerns.

Slowly, I opened my eyes to the SunOS logo. A sigh of relief filled the air conditioned room. The system was coming up properly.

It’s been a couple month since the system had the problem but it hasn’t mysteriously rebooted in that time. We’ve started to recall old tapes from the very beginning of the backup process.

Now that we’re no longer ordering new tapes it’s added a hefty amount towards the backup system that they’ve been apparently planning to implement and just needed budget for. It isn’t much in the grand scheme of the project but the lessons learned added invaluable worth to it all.

Unfortunately until such a time as the new system is in place I still have to go through the motions of using the slow and old robot…

Lose data? Tell me about it (and enter to win a Clickfree backup solution): [link to post] #in

— Posted using Chat Catcher

Good reads/entries in my Backup Horror Stories contest: [link to post] Have you entered yet? Win a Clickfree backup system #in

— Posted using Chat Catcher

Lose data? Tell me about it (and enter to win a Clickfree backup solution): [link to post]

— Posted using Chat Catcher

Makes reinstalls a little easier too.

For anyone who doesn’t mirror their whole drive as part of their backup process, they’ll want to ensure their data is in with the other folders they have flagged for scheduled backups.