Outage Tales: Disgruntled Former Employee Deletes 456 EC2 Instances

Here in the northern hemisphere, the days are getting shorter, the nights are getting longer, and – with Halloween around the corner – it’s the spookiest time of year! In that spirit, we thought we would share some scary stories.

Welcome to Outage Tales, our series of real-life stories about one of the scariest things an enterprise can face: a catastrophic technology outage.

In today’s tale, Arpio’s CEO Doug Neumann tells the story of a disgruntled former employee of Cisco WebEx whose AWS credentials were never revoked. Prepare to be spooked!

Transcript:

Hi, I’m Doug from Arpio, and today I want to tell you the story of Sudish Ramesh.

Sudish is an IT professional who formerly worked for one of the largest hardware manufacturers in the world. He worked on a team collaboration and video conferencing service that they host for their customers in AWS.

Sudhish left this job, and was apparently unhappy upon his departure. Several months later, he discovered that he still had access to their AWS environment, and used that access to terminate 456 of his former employer’s EC2 instances.

Now, that’s a lot of servers to lose, and you probably won’t be surprised to hear that it resulted in a 2-week outage of the service, and it cost 2 and a half million dollars for them to recover. It also landed Sudhish in jail.

So, what should we learn from this tale?

Well, aside from the obvious “don’t forget to delete IAM users when an employee leaves,” it’s worth reflecting on the level of access that is required to work in our cloud environments. In every organization there are some people entrusted with administrative access to the systems – they can’t do their jobs without it.

But that also means that these single points of failure are unavoidable. And even if you have the most reliable team, credentials can still get stolen. It’s impossible to lock down your environment entirely. All you can do is ensure you’ve got a backup solution that can’t be defeated by a bad actor.

At Arpio, our product not only secures your backups, but also your entire recovery environment. If a disgruntled employee, or any bad actor, does something unfathomable, they can’t undermine your ability to recover. And, we can get you back online in minutes.

If you want to talk more about protecting your AWS workloads, contact us at arpio.io.