I guess part of the fun of working as a systems administrator is to recover from a server disaster such as having random parts of your RAID 1 (mirrored hard drives) hard drives erased so that the machine does not boot and personal data be erased. Of course, it would not have been so painful had we had a real backup plan. Unfortunately, I had just started to do some basic backup to a central server when this happened. And further unfortunately, since I was using some old scripts, the backed up data were erased Monday morning as in my old workplace, we would have had the data backed up to tape on Friday. Anyways, at least the critical data did not get erased.

The really fun part was trying to bring our trouble ticket system back online. We use Double Choco Latte (DCL), an open source product, which we had to make many changes in order to make it work with our system. Of course, since this is not documented and the knowledge lives in one person’s head, it was quite a challenge to bring it back online. It goes to show how open source is not always the answer, but it still beats paying $7,000 to get something else that does not work.

So now things are mostly back to normal. We also found out that to recover data from the hard drive would take $10,000+, so since it is not exactly critical data that we need, I think we will be passing on this one. So the lesson of the month is…do your backups!

This entry was posted on Tuesday, August 17th, 2004 at 3:05 pm and is filed under Work. You can follow any responses to this entry through the RSS 2.0 feed. Both comments and pings are currently closed.