Colossal IT Fail: Accidentally formatting hard disks of 9,000 PCs and 490 servers

HOLY CRAP.  Here is an IT horror story from Rod Trent over at myITforum.com (a great site if you get the chance to head there):

A bit of helpful information for the uninitiated:

SCCM or (Systems Center Configuration Manager) from Microsoft is a tool IT administrators use to facilitate automated deployments of patches, software, settings, to collections of computers (perhaps based upon memory capacity, hard disk space, OS version, etc. – a wide etc. while also allowing them to have a very granular inventory of their systems at any one time.

This is a story of SCCM gone wrong…but not because SCCM is bad, mind you, but just like anything…if you don’t know what you are doing or aren’t paying close enough attention, the tool you thought was so useful can be your undoing.

In “Lessons in what NOT to do with SCCM”, we learned that a misguided patch distributed through SCCM may have taken down an entire Australia banking system.  Then, in “Update to the SCCM package heard round the world” we heard about the numbers of desktops and servers effected, along with how HP (to whom CommBank outsources infrastructure services) is scrambling to make amends, sending HP CEO, Meg Whitman, into the fray.

Well, over the weekend, I was able to source even more scoop on this.  No one has been able to get a clear picture of the true issue until now – partly because there are plenty of fingers pointing, and partly because of pride and cover-up.

Of course…as we all know working with SCCM over the years, SCCM picks-up most of the blame when things go wrong.  However, we also know that SCCM simply does exactly what you tell it to do.  Still, it’s an easy target, particularly for those upper management types who really have no clue about how technology works.  They think they do, but when you actually take them to task, they prove their lack of knowledge in 30 seconds or less.  When that happens, it’s best to just snicker under your breath and walk away.  Confronting them with it (particularly in front of others) just gets you a direct ticket to the list of employees to show the door when times get tough.

It has been reported previously that a “patch” was the culprit of the issue at CommBank.  This was a rumor, and if it was indeed a patch, there would be a lot of others besides HP scrambling.  If it was a “patch” you’d see Microsoft onsite right beside the HP brass.  There has been question after question in the communities asking “what patch?”, “which one?”, because if a patch caused the issue it could cause problems in other companies.  Microsoft patches are developed uniquely in that they will not install on a system where it is not compatible or required.  Plus, if it were a Microsoft patch that caused the problem, Microsoft would have been helping the company rollback the errant patch.  So, folks, you can rest easy – this was not a patch.

No…this was the result of Task Sequence distributed to a custom SCCM Collection.  The Collection had been created/modified by an HP Engineer (adding a wildcard) and the engineer had inadvertently altered the Collection so that it was very similar in form and function to the “All Systems” Collection.  The Task Sequence contained automation to – here it comes – format the disks.  Yes, the disks of some 9,000 PCs and 490 servers (including domain controllers) were formatted and wiped clean.

Right now, HP is working night and day to rectify the situation.   And, of course, they (HP) are attempting to blame SCCM, saying it’s SCCM’s fault for not prompting an alert before the wiping out the disks. Anything to shift blame, I guess.  What did I say earlier?  30 seconds or less?

This kind of thing happens to me ALL the time.

I worked at Best Buy back in 2001 and my third day on the floor a lady calls asking how much our modems were. I took a quick walk to the aisle and saw we had them for like $70+ depending on external/internal, brand, etc.

I told her this on the phone and she was ecstatic and said she’ll be right over. This surprised me because even in 2001 these weren’t great prices.

Well about 15 minutes later I’m being paged to the front to meet a very angry lady. She accused me of lying about the prices of modems and how I wasted her time, her gas, blah blah. Turns out she thought the computer was the modem and I stupidly thought a modem was a modem.

Perhaps this person needs to read our PSA.

via: [Reddit\TalesFromTechSupport]

Picture Source: [Richard Masoner / Cyclelicious (CC)]

Tired of using a mouse AND a keyboard? We have an answer for you!

I just don’t know what to think of this thing.  Tell you what, Combimouse: you send me one of these, and I’ll review and write it up for GAS, eh?

That second half you see there acts as the (obviously) right-side of your keyboard, but also as a mouse…I won’t lie, the concept is cool, but I can totally envision Scott hiding the “mouse” half of this on me just to be a jerk.

Seriously, if you are into crazy ways of improving your productivity and don’t care if you get made fun of by the rest of your IT staff (or don’t have a “rest of the IT staff” to make fun of you), then support the Combimouse Kickstarter campaign.

via: [Combimouse]