Twelve years ago I was with my colleague contracted for 24/7 support for some company – easy job until we got a call from the IT guys Friday evening: there was a fire in the server room. We thought the IT guys were joking to invite us for some beer for the weekend in a “funny” way.
We get in the car and make our way there. From far away we saw fire tracks standing before the building; the fire was already over, but the server room was almost completely gone. Our servers had literally melted.
The same evening, from spare parts, we were able to assemble the most crucial server responsible for driving production.
There was a backup policy in place, and in theory, we should have had backups offsite, but there was no strict control of this! So, the only backup we had was on tape in the same server room which had burned. Fortunately, the backup library was on the lowest shelf, and the last full backup was on a tape in the lowest slot (just above the floor), so while temperatures during fire was about 700C in the room, but just under the floor it was much less.
This was the only backup tape which survived.
We tried to restore from it – the tape drive malfunctioned (because the tape was so dirty from fire). Our hardware guys unwound the tape and removed all the debris with alcohol and rewound it. We then attempted a restore from another drive – this failed too. We then repeated the procedure third time and it worked!
We ended up restoring the server and the data. đŸ™‚
-I would bet you had a major DR plan re-write after that, eh? Edited for grammar/spelling – Rob
via: [Mariusz6237 @ Spiceworks Community]
