Doing whatever it takes

Twelve years ago I was with my colleague contracted for 24/7 support for some company – easy job until we got a call from the IT guys Friday evening: there was a fire in the server room. We thought the IT guys were joking to invite us for some beer for the weekend in a “funny” way.

We get in the car and make our way there. From far away we saw fire tracks standing before the building; the fire was already over, but the server room was almost completely gone. Our servers had literally melted.

The same evening, from spare parts, we were able to assemble the most crucial server responsible for driving production.

There was a backup policy in place, and in theory, we should have had backups offsite, but there was no strict control of this!  So, the only backup we had was on tape in the same server room which had burned. Fortunately, the backup library was on the lowest shelf, and the last full backup was on a tape in the lowest slot (just above the floor), so while temperatures during fire was about 700C in the room, but just under the floor it was much less.

This was the only backup tape which survived.

We tried to restore from it – the tape drive malfunctioned (because the tape was so dirty from fire).  Our hardware guys unwound the tape and removed all the debris with alcohol and rewound it.  We then attempted a restore from another drive – this failed too. We then repeated the procedure third time and it worked!

We ended up restoring the server and the data. :)

-I would bet you had a major DR plan re-write after that, eh?  Edited for grammar/spelling – Rob

via: [Mariusz6237 @ Spiceworks Community]

 

 

  • sackes

    Edited for grammar/spelling – Rob

    ” far away we saw fire TRACKS standing” :D

    • MrJake

      Could have been from a Delorean…

      • sackes

        LOL

        • http://thefreewarejunkie.com Rob Dunn

          This is is akin to spending the afternoon cleaning the house, then the wife yells at you for not picking up that last pair of socks…*sigh*

          It was waaay worse before I got to it, I assure you – but, I do like “fire tracks!”

          • sackes

            Feel your pain… btdt ;)

            And yeah it does have a ring….

    • defladamouse

      you beat me to it!

  • http://www.facebook.com/karlpalachuk Karl W. Palachuk

    Tapes don’t fail. Technicians fail.
    :-)