This is what we are all about, amiright? -Rob

—

Head out to client for an urgent server rebuild. Their Exchange server has been having intermittent BSODs with random errors that we’ve been diagnosing for a month to no good effect. We’ve already prepped the client that we’ll come on site, snap a backup, blow away the existing server, rebuild and restore – all told about six hours at usual rates.

I get there, get set up to take a backup. Site boss asks what I’m doing. I tell him, “Taking a backup before I blow away the server.” He informs me they have a backup and says, “I’m going to pay you for two hours of sitting around.” I reply, “If I can’t take a backup, I’m not going to start the migration. That’s an automatic no-go.” He insists it’s not going to happen. I start packing. He calls my boss to tell him what an idiot I am, and my boss… backs me up, telling him it’s our backup runs or the job won’t happen.

This crisis resolved, I start the backup and gather the other information I need while it’s spooling their small amount of data – less than 20GB – to an external 500GB drive.

Blow away server. Install Windows 2003. Install Exchange with the /disasterrecovery switch. Restore from client backu- oh! Wait!

The client’s backups are garbage. They haven’t bought a new tape in over five years, haven’t cleaned the tape drive since it’s birth, and errors abound in the logs.

Spin up my backup, restore data, all is well an hour before deadline.

tl;dr: Never EVER trust the client’s backup system. ALWAYS take a second backup on different media.

EDIT: When I’m running training courses, I always tell them this story. In a long history of server rebuilds and migrations, taking a second backup has saved the client’s data (and my own backside) three times so far.

via: [Spiceworks Community]
Picture Source: [Brett Jordan (CC)]