First off I will say that I work for a very large retail corporation who makes billions annually and has a very large IT department but only a handful work on the actual Network side of things. I’m in the Network category, all 25 or so of us.
I deal with the store side of this company’s network, so most of my time is spent monitoring the physical equipment in all of our store remotely through various polling tools that we have set up. We have 2 rooms in each store that store our equipment (Switches/routers) which are our MDF and IDF. IDF controls anything related to registers or customer facing wifi and such and the MDF controls workstations, DVR, voip phones, etc… So now we can see the picture. Typical network layout. Now I will get into my clusterf$#k of a situation that happened 5 hours ago that seems to have left me still shaking in anger and disbelief.
As I sit at my desk, minding my own business, I get an alert from a store which states that their IDF closet is currently offline. Okay, no big deal right? Probably a power outage, someone tripped over a cable, nothing too serious. So I do what I normally do and call the store and ask to speak to the manager. (Now, keep in mind these people are not IT professionals, at all, so we are very patient with them and do our best to walk them through everything step by step) I eventually get the manager on the phone (we’ll call her Lisa) and the conversation goes like this.
Lisa: hello?
Me: Hi, This is “Steve” from Network
Lisa: Oh thank goodness you’ve called, our registers are offline!
Me: Yes, I see that. Can you please head to your IDF room so we can begin to diagnose the problem?
Lisa: Oh sure!
At this point she seems happy to assist and appears to be at least 10% more competent than a grapefruit, which is a plus for this sort of thing, so I’m thinking this is going to be a short call and I can go on about my business.
Wrong
Lisa: Okay, so I’m in the room now!
Me: Alright, I forgot to ask you before, have you experienced any power outages today? Even brief ones?
Lisa: No, not that I can recall.
Me: Okay, that’s fine. Now the first thing I’m going to have you do is locate our 3 devices which help run the network. They are labeled IDF1, IDF2 and IDF3.
Lisa: I see them!
Me: Great, do they appear to have any power going to them, such as lights, fans running, anything?
Lisa: Uhhhhhh no, I don’t see any lights. Nothing in this room is on. I unplugged everything.
Me: (I freeze in terror. Did I hear her correctly? What exactly does she mean?) Define, “Everything” please.
Lisa: Oh well we’re getting WIFI so I’m making room for the equipment because they said it was going to be installed in this room. Figured I’d start moving everything into another storage room to make room, yaknow to save time. So I started today by just unplugging everything and started packing up the cables.
Me: You have got to be kidding me (At this point almost all professionalism has be lost, but just enough stays so that I can stay on the right side of that fine line that defines employment and unemployment) did you know that you’re not even supposed to get wifi until next year and that this room ran your registers and cash office?
Lisa: Oh no, I thought it was just a room full of old junk. (Cisco3560s) and all the other store are getting wifi now why aren’t we?
Me: Right now wifi is the least of your problems.
I blacked out momentarily from rage.
When I regained consciousness I was able to walk Lisa re-cabling everything and getting everything powered back up (2hrs after I saw the IDF drop). By this time customers were getting pissed because registers were down and the store associates didn’t know how to handle this madness without their beloved manager (who happened to be the only one on at the moment) so in the end chaos was neutralized and lessons were learned. I just hope this never happens again and if it does may the call go to another unfortunate soul.
via: [Reddit\TalesFromTechSupport]
Picture Source: [Jemimus]
– I momentarily blacked out from rage! First lesson – – tell IT EVERYTHING you did, no matter how trivial you think it is. It will save you trouble in the long run. Will we make fun of you after we get off the phone? More than likely. We will make fun of you even more if you don’t tell us everything. -Rob
