Background: I work on the support desk for a niche, business-only ISP. Our broadband products are expensive because you get direct support to people like me.
The Customer: A high end sales floor that sells 200,000 dollar yachts and motorhomes.
The IT Manager for The Customer: Got the job because “his brother used to do it”.
So last week, they upgraded their broadband from ADSL to a newer VDSL setup with us. All was good for 2 days (in which they downloaded 50GB of data) and then on Monday phoned in furious that they had no connection.
On inspection, I found the connection to be up and running. I could telnet to their router, and could even see traffic passing through it. After talking to their IT manager, he told me they seemed to be unable to browse to anything for 10 mins, then it’d work for 5 mins, then off again. He assured me he’d checked their LAN and servers and could find nothing wrong.
Should I mention that their LAN IP addressing is using public IP addresses that don’t belong to them and are NAT’d? Yeah. Red flag there.
Our network engineers could find nothing wrong, and I received a lovely missive from a California University telling me that The Customer’s real public IP was trying to hack into remote desktop servers. Lovely.
I tell The IT Manager. First he demands I tell him how to find the virus (wut?). I suggest he uses his antivirus, then things like netstat from a DOS prompt etc. He calls back an hour later, absolutely furious. He insists there’s no virus on their network. He insists everything is perfect. He is absolutely adamant that it’s the broadband line.
We send out a field engineer. What does he find?
- Two Windows 2003 servers riddled with viruses (running DHCP, DNS and basic file sharing – no domain or permissions).
- Two Linux servers for which noone has passwords and noone even knows what they’re doing.
- About 40 workstations, all running pirated copies of XP, also jammed with nastiness. Everyone is full admin on their local machines AND all the network shares.
- A hodge-podge of network gear by different vendors, none of which they have passwords for.
- No backups of any of their data (not that they didn’t have an online backup account provided by us or anything… The IT Manager just never bothered to set it up).
- No antivirus installed on any machine. We found The IT Manager wandering around with a USB stick with a portable copy of AVG Free Personal edition on it, plugging it into each machine one at a time and making a fantastic effort of spreading around the viruses even more while fixing nothing.
Unplugging their shithole of a network from the router results in a perfect, constant 40Mbps.
It’s now Wednesday. We’ve had an engineer on site for two days straight trying to fix their broken computers and network. Our poor field engineer had no idea where to start, and contemplated telling them to set fire to the place and take the insurance money, as literally every single thing he looks at to do with the computers or network is either somehow broken, misconfigured, or pirated. Their IT manager has gone AWOL (didn’t show up for work today), and we’re quoting them for a full service contract.
If I ever meet the man, I think I might punch him for calling himself an IT manager.
via: [Reddit\TalesFromTechSupport]
Picture Source: [joestump (CC)]
