We were in the beginning phases of implementing a new version of our EMR system, which required substantial network upgrades to accommodate high-availability (i.e. if one server went down, traffic re-routed to another in a different building).  As part of this upgrade, we needed to completely gut a switch closet at the secondary site (a site that serves as our surgery facility).  This upgrade also required synchronized work to be done at our core switch so we could update the network configuration and move the WiFi access points around.

So, two of us come in on a Friday evening at 7:00pm and completely remove all the wiring, network hardware, battery backups, two-post rack, etc., basically meaning we were rebuilding from nearly scratch.  Not a standard task for us, that’s for sure.

We assembled a new 4-post rack, performed some network configurations, placed new battery backups, two additional servers and a small array of network devices and got everything back up and running without a hitch.  It wasn’t an easy or short process, by any means, but it was done.  To our behest, the satellite TV for the lobby was hooked into our network power, so it also had to be unplugged and reconnected…not our stuff, but we moved it anyway.  We couldn’t really test it, but we plugged it in EXACTLY as we had found it…we figured we could get someone in on Monday to look at it if it wasn’t working right.  It was 2am, so we figured we’d go home.

Emails went out to the manager stating the cable was something we could not test and it was plugged into our rack power, here’s an FYI…blah blah. I was a little concerned about what Monday would bring, as this manager has always been a thorn in our side as she tended to be very passive-aggro and didn’t understand technology at all.

Monday arrives, the manager EXPLODES and tells everyone she can think of (including our CEO) that the television is down in her lobby and how “IT majorly $#@%ed everything up here.”  Mind you, we had no firsthand response from her over the weekend or Monday.  I didn’t find out that she was upset until after I called to the facility first thing in the morning to find out how everything was going.  It goes up and down the chain and eventually I respond to her through my boss with a description of what had gone on.  Basically – “We completely rebuilt your network from the ground up in a few hours and we are getting yelled at over the TV – after we let her know about it.”

As it turned out, the wiring for the unit was completely wrong – it had been from the start.  It was only by a fluke that it was working at all before it was shut off and turned back on.

I got a very awkward apology by early afternoon from the manager.

[Picture Source: Kay Kim(김기웅) (CC)]