To Kill A Server: The Time-Bomb Email [story]

2
1128

I used to work at a university, doing general IT-stuff, such as cleaning out data, shouting at people to clean out their personal account, retrieving “lost” data, etc. etc.

One fateful day, an unnamed faculty member decided to use her contacts to promote her friends baby in some popularity competition, thinking to use the rougly 8000 students and faculty at the universe to garner some e-votes. This is against policy, but not terrible in itself. Her method, however, was much worse than her intentions. Here is a short timeline of the events.

  • At 3 am, Timebomb-Mary sent an email containing a 200kb picture and several lines of text to every single mailing list and contact she could find on her account.
  • Around 8.30am, people check their email. 7988 people roll their eyes and delete the email. A dozen people responding with various forms of “I can’t believe you did this” using the reply-to-all button. People demanded everyone stop replying-to-all… using reply-to-all. Of course, students being students, things got out of hand quickly.
  • At 9.30am, thanks to Exchange quoting the previous email on reply, and helpfully including all previous email addresses, single emails started reaching 12mb each, one single reply generating almost 10gb of data.
  • At 9.35am, every single mailbox on campus was full, and generating 8000 “This person cannot recieve your email” messages per reply. 30 seconds later, the overflow to prevent people actually losing email was also full.
  • At 9.38am, the exchange server gave up.

The email server had to be taken down, the emails deleted and severe punishment was threatened for those who would use the timebomb email again. 14 TB of spam was deleted, over 150.000 messages were lost due to full mailboxes, an unknown number of which were actually relevant. 180 accounts were semi-wrongfully quarantined for sending spam, before that system too ground to a screeching halt.I never did find out what happened to the person who started this, but I can only hope it was painful.

Leave it to users, they break everything sooner or later. -Scott
BackBlaze