Tag Archives: incident response

DOJ Best Practices for Breach Response

Posted on by
3

I stumbled on this great release from the US Department of Justice – a best practices guide to breach response.

Reading it is rather reminiscent of much of what we said in the 80/20 Rule of Information Security years ago. Namely, know your own environment, data flows, trusts and what data matters. Combine that with having a plan, beforehand, and some practice – and you at least get some decent insights into what your team needs and is capable of handling. Knowing those boundaries and when to ask for outside help will take you a long way.

I would also suggest you give our State of Security Podcast a listen. Episode 6, in particular, includes a great conversation about handling major breaches and the long term impacts on teams, careers and lives.

As always, if we can assist you in preparing a breach response process, good policies, performing those network mappings or running table top exercises (or deeper technical red team exercises), let us know. We help companies around the world master these skills and we have plenty of insights we would love to share!

First Step After Breach

Posted on by
Reply

Discovering an information security breach can be a shock! Picture it: you are enjoying a regular work day and WHAM! Suddenly you are at the center of an incident that could possibly affect the future of the company and perhaps your own future as well. It’s easy to panic. You know if you don’t do the right thing, right now, bad things are sure to rain down on you. So, what is the very first thing that you should do?

Go immediately to your incident response plan, of course! After all, that is the reason your company has put together an IR plan and team in the first place; to plan for contingencies so that personnel don’t go off half-cocked and lose vital data and evidence. 

But is your plan clear enough that regular system users or even help desk personnel know what to do first without having to thumb through a hundred pages of plan? If not, perhaps a simple little trick we use in our incident response plans will work for you. 

The very first thing you see when you open one of our incident response plans are employee and incident response team Quick Response Guides (see the example of an employee guide below-the IRT guide is similar, but more complex). 

I know from my military experience that having checklists such as the Quick Response Guides in place truly cuts down on mistakes and helps calm personnel during difficult situations. Why not see if they can also improve your response quality?

 

Chart

 













You can download the pocket guide here. 

Thanks to John Davis for this post.

The Need for an Incident Recovery Policy (IRP)

Posted on by
Reply
Organizations have been preparing for information security issues for a number of years now and many, if not most, have embraced the need for an incident response policy and process. However, given the recent spate of breaches and compromises that we have analyzed and been involved in over the last year, we have seen an emerging need for organizations to now embrace a new kind of policy – a security incident RECOVERY policy.
 
This policy should extend from the incident response policy and create a decision framework, methodology and taxonomy for managing the aftermath of a security incident. Once the proverbial “fire has been put out”, how do we clean up the mess, recreate the records we lost, return to business as usual and analyze the impacts all of this had on our operations and long term bottom line? As a part of this process, we need to identify what was stolen, who the likely benefactors are, what conversion events have taken place or may occur in the future, how the losses impact our R&D, operational state, market position, etc. We also need to establish a good working model for communicating the fallout, identified issues, mitigations, insurance claims, discoveries and lessons learned to stakeholders, management, customers, business partners and shareholders – in addition to the insurance companies, regulators and law enforcement.
 
As you can imagine, this can be a very resource intensive process and since post-incident pressues are likely to remain high, stress levels can be approaching critical mass and politics can be rampant, having a decision framework and pre-developed methodology to work from can be a life saver. We suggest following the same policy development process, update timeframes and review/practice schedules as you do for your incident response policy.
 
If your organization would like assistance developing such a policy, or would like to work through a training exercise/practice session with an experienced team, please feel free to work with your account executive to schedule such an engagement. We also have policy templates, work sheets and other materials available to help with best practice-based approaches and policy creation/reviews.

The Big Three Part 3: Incident Response

Posted on by
Reply

It’s been a couple of busy months since we posted parts one and two of this series, so I’ll recap briefly here. Part one talked about the failure of information security programs to protect private data and systems from compromise. It showed that despite tighter controls and better security applications, there are more data security compromises now than ever. This was the basis for suggesting an increased emphasis on incident detection, incident response and user education and awareness; the Big Three.

Part two in the series discussed information security incident detection and how difficult it is to implement effectively. It related the sad statistic that less than one out of five serious data breaches is detected by the organization affected, and that a disturbing number of breaches go undetected for months before finally being uncovered. Part two also touted a combination of well configured security tools coupled with human monitoring and analysis as one of the best solutions to the problem. In this installment, we’ll discuss the importance of accompanying incident detection with an effective, well-practiced incident response plan.

Say that an ongoing malware attack on your systems is detected, would your staff know just what to do to stop it in its tracks? If they don’t do everything quickly, correctly and in the right order, what could happen? I can think of a number of possibilities right off the bat. Perhaps all of your private customer information is compromised instead of just a portion of it. Maybe your customer facing systems will become inoperable instead of just running slow for a while. Possibly your company will face legal and regulatory sanctions instead of just having to clean up and reimage the system. Maybe evidence of the event is not collected and preserved correctly and the perpetrator can’t be sued or punished. Horrible consequences like these are the reason effective incident response is increasingly important in today’s dangerous computing environment.

Developing and implementing an incident response plan is very much like the fire drills that schools carry out or the lifeboat drills everyone has to go through as part of a holiday cruise. It is really just a way to prepare in case some adverse event occurs. It is deconstructing all the pieces-parts that make up security incidents and making sure you have a way to deal with each one of them.

When constructing your own incident response plan, it is wise to go about it systematically and to tailor it to your own organization and situation. First, consider the threats your business is menaced by. If you have been conducting risk assessments, those threats should already be listed for you. Then pick the threats that seem the most realistic and think about the types of information security incidents they could cause at your organization. These will be the events that you plan for.

Next, look over incident response plans that similar organizations employ and read the guidance that is readily available our there (just plug “information security incident response guidelines” into a web browser and see what you get – templates and implementation advice just jump off the page at you!). Once you have a good idea of what a proper incident response plan looks like, pick the parts that fit your situation best and start writing. This process produces the incident response policies needed for your plan.

After your policies are set, the next step I like to tackle is putting together the incident response team. These individuals are the ones that will have most of the responsibility for developing, updating and practicing the incident response procedures that are the meat of any incident response plan. Armed with the written policies that were developed, they should be an integral part of deciding who does what, when it gets done, where they will meet, how evidence is stored, etc. Typically, an incident response team is made up of management personnel, security personnel, IT personnel, representative business unit personnel, legal representatives and sometimes expert consultants (such as computer forensics specialists).

Once all the policies, personnel and procedures are in place, the next (and most overlooked part of the plan) is regular practice sessions. Just like the fire drills mentioned above, if you don’t actually practice the plan you have put together and learn from the results, it will never work right when you actually need it. In all my time doing this sort of work, I have never seen an incident response practice exercise that didn’t expose flaws in the plan. We recommend picking real-world scenarios when planning your practice exercises and surprising the team with the exercise just as they would be in an actual event.

In the fourth and final installment of this series, we will discuss user education and awareness – another vital component in recognizing and fighting data breaches and system security compromises. 

Thanks to John Davis for this post.

Monitoring: an Absolute Necessity (but a Dirty Word Nonetheless)

Posted on by
3

There is no easier way to shut down the interest of a network security or IT administrator than to say the word “monitoring”. You can just mention the word and their faces fall as if a rancid odor had suddenly entered the room! And I can’t say that I blame them. Most organizations do not recognize the true necessity of monitoring, and so do not provide proper budgeting and staffing for the function. As a result, already fully tasked (and often times inadequately prepared) IT or security personnel are tasked with the job. This not only leads to resentment, but also virtually guarantees that the job is will not be performed effectively.

And when I say human monitoring is necessary if you want to achieve any type of real information security, I mean it is NECESSARY! You can have network security appliances, third party firewall monitoring, anti-virus packages, email security software, and a host of other network security mechanisms in place and it will all be for naught if real (and properly trained) human beings are not monitoring the output. Why waste all the time, money and effort you have put into your information security program by not going that last step? It’s like building a high and impenetrable wall around a fortress but leaving the last ten percent of it unbuilt because it was just too much trouble! Here are a few tips for effective security monitoring:

  • Properly illustrate the necessity for human monitoring to management, business and IT personnel; make them understand the urgency of the need. Make a logical case for the function. Tell them real-world stories about other organizations that have failed to monitor and the consequences that they suffered as a result. If you can’t accomplish this step, the rest will never fall in line.
  • Ensure that personnel assigned to monitoring tasks of all kinds are properly trained in the function; make sure they know what to look for and how to deal with what they find.
  • Automate the logging and monitoring function as much as possible. The process is difficult enough without having to perform tedious tasks that a machine or application can easily do.
  • Ensure that you have log aggregation in place, and also ensure that other network security tool output is centralized and combined with logging data. Real world cyber-attacks are often very hard to spot. Correlating events from different tools and processes can make these attacks much more apparent. 
  • Ensure that all personnel associated with information security communicate with each other. It’s difficult to effectively detect and stop attacks if the right hand doesn’t know what the left hand is doing.
  • Ensure that logging is turned on for everything on the network that is capable of it. Attacks often start on client side machines.
  • Don’t just monitor technical outputs from machines and programs, monitor access rights and the overall security program as well:
  • Monitor access accounts of all kinds on a regular basis (at least every 90 days is recommended). Ensure that user accounts are current and that users are only allocated access rights on the system that they need to perform their jobs. Ensure that you monitor third party access to the system to this same level.
  • Pay special attention to administrative level accounts. Restrict administrative access to as few personnel as possible. Configure the system to notify proper security and IT personnel when a new administrative account is added to the network. This could be a sign that a hack is in progress.
  • Regularly monitor policies and procedures to ensure that they are effective and meet the security goals of the organization. This should be a regular part of business continuity testing and review.
Thanks to John Davis for writing this post.

Incident Response: Are You Ready?

Posted on by
9

All of us suffer from complacency to one extent or another. We know intellectually that bad things can happen to us, but when days, months and years go by with no serious adverse incidents arising, we tend to lose all visceral fear of harm. We may even become contemptuous of danger and resentful of all the resources and worry we expend in aid of problems that never seem to manifest themselves. But this is a dangerous attitude to fall into. When serious problems strike the complacent and unprepared, the result is inevitably shock followed by panic. And hindsight teaches us that decisions made during such agitated states are almost always the wrong ones. This is true on the institutional level as well.

During my years in the information security industry, I have seen a number of organizations founder when struck by their first serious information security incident. I’ve seen them react slowly, I’ve seen them throw money and resources into the wrong solutions, and I’ve seen them suffer regulatory and legal sanctions that they didn’t have to incur. And after the incident has been resolved, I’ve also seen them all put their incident response programs in order; they never want to have it happen again! So why not take a lesson from the stricken and put your program in order before it happens to your organization too? Preparing your organization for an information security incident isn’t really very taxing. It only takes two things: planning and practice.

When undertaking incident response planning, the first thing to do is to examine the threat picture. Join user groups and consult with other similar organizations to see what kinds of information security incidents they have experienced. Take advantage of free resources such as the Verizon Data Breach Reports and US-CERT. The important thing is to limit your serious preparations to the top several most credible incident types you are likely to encounter. This streamlines the process, lessens the amount of resources you need to put into it and makes it more palatable to the personnel that have to implement it. 

Once you have determined which threats are most likely to affect your organization, the next step is to fully document your incident response plan. Now this appears to be a daunting task, but in reality there are many resources available on the Internet that can help guide you through the process. Example incident response plans, procedures and guidance are available from SANS, FFIEC, NIST and many other reputable organizations free of charge. I have found that the best way to proceed is to read through a number of these resources and to adapt the parts that seem to fit your particular organization the best. Remember, your incident response plan is a living document and needs to reflect the needs of your organization as well as possible. It won’t do to simply adopt the first boiler plate you come across and hope that it will work.

Also, be sure that your plan and procedures contain the proper level of detail. You need to spell out things such as who will be on the incident response team, their individual duties during incidents, where the team will meet and where evidence will be stored, who should be contacted and when, how to properly react to different incidents and many other details. 

The next, and possibly the most important step in effective incident response is to practice the plan. You can have the most elegantly written security incident response plan in the world, and it is still doomed to fail during an actual incident if the plan is not practiced regularly. In all my years of helping organizations conduct their table top incident response practice sessions, I have never failed to see the process reveal holes in the plan and provide valuable lessons for the team members who participate. The important thing here is to pick real-world incident scenarios and to conduct the practice as close to the way it would actually occur as possible. We like to only inform a minimum number of response personnel in advance, and surprise the bulk of responders with the event just as it would happen if it were real. Of course there is much more to proper incident response planning and practice than I have included here. But this should start your organization along the right path. For more complete information and help with the process, don’t hesitate to contact your MSI representative. 

Thanks to John Davis for writing this post.

Incident Response: Practice Makes Perfect

Posted on by
2

 

Is it possible to keep information secure? Read on to find out.

IF there is only one person that knows the information, IF that person never writes that information down or records it electronically, and IF that person is lucky enough not to blurt out the information while they are sleeping, drugged or injured, then the answer is yes…probably. Under any other conditions, then the answer is an emphatic NO! It is an unfortunate truth that no system ever developed to protect the security of information is perfect; they all can be breached one way or another. That is why it is so important to have a good incident response program in place at your organization.

And most of you out there, I’m sure, have an incident response plan in place. All information security standards organizations such as ISO and NIST include incident response in their guidance, and many of you are required to have incident response programs in place in order to comply with regulation. But how many of you practice responding to incidents to make sure your planning actually works? At MicroSolved, we’ve been involved in reviewing, developing and testing information security incident response programs for many years. And we have found that no matter how good response plans looks on paper, they’re just not effective if you don’t practice them. Practicing doesn’t have to be a big chore, either. We’ve helped many organizations conduct table top incident response exercises and they usually only last a few hours. They’ve never failed to produce valuable returns.

Unfortunately, there are no good incident response exercise frameworks available out there – we’ve looked. But it is not hard to create your own. Simply pick a type of incident you want to practice – a malware attack for example. You imagine what such an attack would look like to your help desk personnel, system administrators, security personnel, etc. and construct a scenario from that. You just need a basic outline since the details of the response will construct themselves as you proceed with the exercise.

What we have found from conducting and observing these exercises is that problems with the written plan are always exposed. Sure, maybe the plan says that this group of people should be contacted, but is there a procedure for ensuring that list is always kept current in place? Have you made pre-arrangements with a forensic specialist in case you need one? Are the help desk personnel and desk top administrators trained in how to recognize the signs of an attack in process? These are the types of issues performing simple table top incident response exercises will reveal.

Perhaps you will be lucky and never experience a bad information security incident. But if you do, you will be very glad indeed if you have a well practiced information security incident response program in place!

Responding to a Compromised System Alert

Posted on by
Reply

Thanks to the data from the HITME, I interact with a lot of people and organizations that have compromised machines. Often, my email or phone call is the first they have heard of the problem. Reactions vary from shock and denial to acceptance and occasionally rage. Even worse, when they hear that their machines are attacking others or being used in active attacks, many have no idea how to handle the situation.

Should you ever get a call like this from me or someone else, here are a few tips that you might find helpful for proceeding.

1. Be polite. I am calling to help you. Even though my message may mean more work and possibly some pain for you and your staff, knowing about a compromise is MUCH better than not knowing. Usually, the more polite and nice you are, the more information I will help you understand. I can usually point you in the right direction to begin to understand the issue, but if you act like a jerk, I will likely leave you to it.

2. Begin an investigation as soon as possible. Invoke your incident response process. If you don’t have one, ask for help, or retain assistance. But, please, treat a caller who explains and demonstrates that you have a system compromise with immediate attention. I see hundreds of compromised systems a day and I don’t have time to beg and plead with you to reduce your risk and the risk your systems present to others. I am happy to substantiate my claims, but after I notify you, TAKE ACTION. The majority of compromised systems involved in notification remain under attacker control for extended periods. Often, weeks and months pass by before any apparent action (such as mitigation or clean up) takes place.

3. Do a thorough job of mitigation. I would say that more than 25% of the time (I just started formally tracking this to gather better metrics.) when a site goes through “clean up”, they end up compromised again and right back where they started from. Likely many of these machines are simply bot-infected and the bots just place their malware back on the system after “clean up” is done. Removing the basic tag files or malware, but not understanding how they got there in the first place and fixing that is pretty much meaningless. For example, I have been working with a site presently that has been used as a PHP RFI verification tag file host for weeks. They have “cleaned up” every day for several weeks to no avail. Every night, they get hit by another PHP RFI scanner and it exploits their system and drops a new tag or malware bot. I have tried explaining no less than 10 times how they need to identify the underlying PHP issue, harden the PHP environment (yeah, I sent them the settings) to no avail. This is an example of how to fail at risk, threat and vulnerability management. Don’t do it. Fix the real problems. If you don’t know how, ask and then follow the guidance provided. If you need more help, either retain it or get a scanner and start hardening.

4. Respect the law. Don’t beg me not to turn this over to law enforcement. I have to. I want to, if you are critical infrastructure or some other member of the high threat club. Fix your stuff and manage security appropriately if you’re a member of the club; or you deserve to explain to law enforcement why you declined. Either way, I am going to try and help you and everyone by making the report.

5. List a contact for security issues on your site. Please, when I do call, I need to know who to talk to. At the very least, let your reception folks know how to handle security calls. The last thing you want is for the attacker to continue to compromise your systems while I play in “Voicemail-Land” forever. Remember, help me help you.

Lastly, even if you don’t get this call, do your due diligence. Make sure that your systems are secure and that you have security processes in place. Retain someone to help you manage risk and perform validation. Work with them to create effective risk management techniques for your organization. Hopefully, you won’t be on the other end of the line tomorrow or the next day as I make my round of calls….

If you have any additional suggestions or comments on this approach, please feel free to drop a comment below. As always, thanks for reading and be careful out there.