Dealing With Breaches

Posted on July 31, 2014 in Security • 6 min read

Security is hard. Much too hard to treat effectively in any one blog post or even one book. It’s a complicated, multifarious discipline that combines a host of observational and technical sub-disciplines. The same is true, to a lesser degree, for breach management. On the one hand, a large componenent of dealing with breaches is straightforward legwork, but there are many hidden pitfalls that can reach up and grab the independent operator or small business owner who is insufficiently practiced in the most important art of security: paranoia.


I am an IT professional, a systems infrastructure architect, and a backend software developer. I am not god. I am also not a lawyer. All of the opinions contained below are my opinions based on my education and my work experience, not necessarily a path to true perfection. They are opinion, use them to inform your actions at your own peril. Especially that last paragraph where I talk about your legal obligations. That’s just me parroting what I’ve been told by lawyers.

Lock the known hole

The first think you need to do in any security breach situation is to lock the known hole. Whatever door that was not supposed to be opened that was opened in order to gain illegitimate access to your systems, lock it and double or quadruple the amount of attention being paid to it immediately. Did one of your systems contract a virus? Unplug it from the network immediately. Put it in quarantine. One of your users gave up their account password? Change the password on their account, and if at all possible lock them out until further notice. The first thing you must do is make sure nothing worse is going to happen.

You may perhaps be asking “Why is he being so paranoid about this, when the damage is already done?” This is sheer foolhardiness. if you came home to your door busted in and blatant symptoms of forced entry strewen all across your living-room, would you assume that there was nobody in your house? Whoever broke in is at the very least a criminal. The only safe approach is to presume that they do not have scruples or morals that they are bringing to the situation, and that they either haven’t left yet or will be back for more.

2ndary infections

Somebody has broken into your home or business. You checked the house, and nobody else is in it right now. You locked and bolted the door and put a chair in front of it just to be safe. What next? Well if you’re not a complete fool, you call the cops. Or somebody like me, in the digital analogy. Don’t move anything, don’t touch anything, don’t go hunting for the problem or deleting all of your “infected” emails. You will miss things and possibly destroy important evidence.

But let’s say you’re a fool, or you just want to know more about my basic process. Well, at this point, you need to start worrying about persistent threats. When a burglar breaks into your house, there is no guarantee that he does not take pictures of your important documents, credit cards, and make copies of your keys so that he can re-enter later. Will your garden variety burglar go through this effort? Probably not. But take the pain of losing everything and divide it by that likelihood, and I find you typically still end up with a troublingly large sum.

So, you need to assume that while your system was breached, everything that the attacker had access to was copied. If you gave up your email password, assume that every single email that you can obtain access to via that account has been copied. Assume every account that was associated with that email was compromised. Why? If I had access to your email account and wanted to extract maximal profits, the very first thing I’d do is start changing the passwords on all of your accounts. While I had access I’d be able to then change the email addresses of record to accounts that you have no control over.

What’s the end result of this on the recovery procedure? A lot of boring, nitpicky legwork. You need to trawl through all of the emails (a local copy pre-dating the breach if you have it, because there is no guarantee that the attacker has not deleted emails of interest after he found them), find every account or subsystem that could have been compromised, and sanitize and re-secure that as well. As well as every sub-system of that system, until you run out. More than likely, it will be more efficient to simply re-secure every system you use, because it’ll take just as long figuring out what is compromised vs. what isn’t.

I like to think of it as a giant, branching, disastrous tree of suck.


While it’s one thing that your security got breached (that can happen to any entity given a sufficiently skilled, determined, and well-financed adversary), it’s another thing to get breached and not know about it right away. A large percentage of breaches are given away not by some sort of intelligent monitoring, but by people noticing the usage of accounts in strange ways. Perhaps one of your employees starts spewing spam like a fountain, or perhaps you suddenly have no bandwidth as all of your business records get exported over your network. The fact that you got taken in such a way indicates not only that you had insufficient security, but that you had insufficient monitoring and self-surveilliance. As a business especially, you are a many handed entity, and you need to know what each and every one of those hands is doing and when it’s behaving in an irregular manner.

So in addition to patching up all the security, locking all the doors with brand new keys, you need to implement new monitoring systems to make sure that, in the future, you can notice potential problems before they become real problems. E-mails should have rate-limiting, and should send alerts when the rate limits are exceeded. Networks too. And it’s actually surprisingly easy to tell the difference between “human-like” and “machine-like” behavior on a network. Most (but not all of course) malicious code is designed to steal data as rapidly as possible, much faster than humans ever operate. Looking for these telltales is easy, and will help catch breaches much earlier in the future.


So, you’ve got everything cleaned up, every nook and cranny scrubbed. You’re done, because you now feel warm, safe, and secure. Now all you have to worry about is your other obligations. And speaking of your other obligations, here’s a new one to add to your stack: required disclosure.

Although the United States federal government does not have laws that cearly require breach disclosure, many states do. Additionally, many foreign nations (I’m not from them, which makes them foreign to me, apologies) have their own disclosure requirements as well. Obviously I cannot cover every disclosure law here (although I may go over some of them at some later date) but the fact remains that you are likely legally obligated to make some disclosure to some individual after almost every breach. If you think that you are not required to make a disclosure to some individual or entity, you likely aren’t considering all of the subsystems affected by your breach. Just because an email breach only gives away the email of your users doesn’t mean that you haven’t given away information in those emails.

If you think that you are not required to make a disclosure to at least some individual or entity, you likely aren’t considering all of the subsystems affected by your breach. Just because an email breach only gives away the email of your users doesn’t mean that you haven’t given away information in those emails.

Or their attachments. Did somebody check all of those PDFs to make sure there wasn’t anything important in them? No. Congratulations, you get to go back to step 2.

Feel free to drop me a line if you want some help!

Jason R.