Incident Response: Practice Makes Perfect

 

Is it possible to keep information secure? Read on to find out.

IF there is only one person that knows the information, IF that person never writes that information down or records it electronically, and IF that person is lucky enough not to blurt out the information while they are sleeping, drugged or injured, then the answer is yes…probably. Under any other conditions, then the answer is an emphatic NO! It is an unfortunate truth that no system ever developed to protect the security of information is perfect; they all can be breached one way or another. That is why it is so important to have a good incident response program in place at your organization.

And most of you out there, I’m sure, have an incident response plan in place. All information security standards organizations such as ISO and NIST include incident response in their guidance, and many of you are required to have incident response programs in place in order to comply with regulation. But how many of you practice responding to incidents to make sure your planning actually works? At MicroSolved, we’ve been involved in reviewing, developing and testing information security incident response programs for many years. And we have found that no matter how good response plans looks on paper, they’re just not effective if you don’t practice them. Practicing doesn’t have to be a big chore, either. We’ve helped many organizations conduct table top incident response exercises and they usually only last a few hours. They’ve never failed to produce valuable returns.

Unfortunately, there are no good incident response exercise frameworks available out there – we’ve looked. But it is not hard to create your own. Simply pick a type of incident you want to practice – a malware attack for example. You imagine what such an attack would look like to your help desk personnel, system administrators, security personnel, etc. and construct a scenario from that. You just need a basic outline since the details of the response will construct themselves as you proceed with the exercise.

What we have found from conducting and observing these exercises is that problems with the written plan are always exposed. Sure, maybe the plan says that this group of people should be contacted, but is there a procedure for ensuring that list is always kept current in place? Have you made pre-arrangements with a forensic specialist in case you need one? Are the help desk personnel and desk top administrators trained in how to recognize the signs of an attack in process? These are the types of issues performing simple table top incident response exercises will reveal.

Perhaps you will be lucky and never experience a bad information security incident. But if you do, you will be very glad indeed if you have a well practiced information security incident response program in place!

Responding to a Compromised System Alert

Thanks to the data from the HITME, I interact with a lot of people and organizations that have compromised machines. Often, my email or phone call is the first they have heard of the problem. Reactions vary from shock and denial to acceptance and occasionally rage. Even worse, when they hear that their machines are attacking others or being used in active attacks, many have no idea how to handle the situation.

Should you ever get a call like this from me or someone else, here are a few tips that you might find helpful for proceeding.

1. Be polite. I am calling to help you. Even though my message may mean more work and possibly some pain for you and your staff, knowing about a compromise is MUCH better than not knowing. Usually, the more polite and nice you are, the more information I will help you understand. I can usually point you in the right direction to begin to understand the issue, but if you act like a jerk, I will likely leave you to it.

2. Begin an investigation as soon as possible. Invoke your incident response process. If you don’t have one, ask for help, or retain assistance. But, please, treat a caller who explains and demonstrates that you have a system compromise with immediate attention. I see hundreds of compromised systems a day and I don’t have time to beg and plead with you to reduce your risk and the risk your systems present to others. I am happy to substantiate my claims, but after I notify you, TAKE ACTION. The majority of compromised systems involved in notification remain under attacker control for extended periods. Often, weeks and months pass by before any apparent action (such as mitigation or clean up) takes place.

3. Do a thorough job of mitigation. I would say that more than 25% of the time (I just started formally tracking this to gather better metrics.) when a site goes through “clean up”, they end up compromised again and right back where they started from. Likely many of these machines are simply bot-infected and the bots just place their malware back on the system after “clean up” is done. Removing the basic tag files or malware, but not understanding how they got there in the first place and fixing that is pretty much meaningless. For example, I have been working with a site presently that has been used as a PHP RFI verification tag file host for weeks. They have “cleaned up” every day for several weeks to no avail. Every night, they get hit by another PHP RFI scanner and it exploits their system and drops a new tag or malware bot. I have tried explaining no less than 10 times how they need to identify the underlying PHP issue, harden the PHP environment (yeah, I sent them the settings) to no avail. This is an example of how to fail at risk, threat and vulnerability management. Don’t do it. Fix the real problems. If you don’t know how, ask and then follow the guidance provided. If you need more help, either retain it or get a scanner and start hardening.

4. Respect the law. Don’t beg me not to turn this over to law enforcement. I have to. I want to, if you are critical infrastructure or some other member of the high threat club. Fix your stuff and manage security appropriately if you’re a member of the club; or you deserve to explain to law enforcement why you declined. Either way, I am going to try and help you and everyone by making the report.

5. List a contact for security issues on your site. Please, when I do call, I need to know who to talk to. At the very least, let your reception folks know how to handle security calls. The last thing you want is for the attacker to continue to compromise your systems while I play in “Voicemail-Land” forever. Remember, help me help you.

Lastly, even if you don’t get this call, do your due diligence. Make sure that your systems are secure and that you have security processes in place. Retain someone to help you manage risk and perform validation. Work with them to create effective risk management techniques for your organization. Hopefully, you won’t be on the other end of the line tomorrow or the next day as I make my round of calls….

If you have any additional suggestions or comments on this approach, please feel free to drop a comment below. As always, thanks for reading and be careful out there.