For this third installment in my series on human-based information security I will be discussing the idea that human analysis and interaction must be at the core of any truly effective monitoring program. To reiterate the basic point of this series, it is axiomatic that information security is a human problem, not a technological problem. It is my contention that our failure to embrace this truth is demonstrated in the fact that not only are more data breaches and other security failures still occurring, they are actually becoming more prevalent over time.

Largely because of this increase in data breaches, and the public outcry about them, incident response has become an increasingly important part of the information security effort over recent years. And as anyone who has actually participated in the effort knows, the most difficult part of incident response is detection. Most network compromises go unnoticed for days, weeks, months, even years! And the only way that l know of to address this problem is through verbose system logging and effective monitoring processes.

System logging and monitoring has always been a thorn in the side of every CISO and network security admin. It seems like such an overwhelming task that most of them automatically give up and either hardly log and monitor at all, or turn the effort over to a third-party service provider that will never be as invested in the task as they would be in their own company’s security monitoring. We always go that extra mile to protect our own, don’t you think?

However, this task isn’t nearly as daunting as it seems. With the help of proper parsing and aggregating tools and a handful of command line tools, the task can be reduced to the human scale. Personnel can pull just that information they need from these mounds of data, and are much better at recognizing anomalies and danger signs than any program, especially over time. It is also a happy fact that the more you perform log searching and monitoring, the easier it is and the better you get at it. One particularly good log monitoring engineer I know even gleans valuable information by simply scrolling through the raw log data at high speed; he is able to see areas of concern in the data through pattern recognition alone.

The caveat of all this is that you need to allocate dedicated personnel to perform these tasks, and these personnel need to be very well trained and knowledgeable about the full capabilities of the tools they use to parse, aggregate and monitor the log data generated by the system. This means proper support and funds allocation at the management level. Our job as information security professionals is to ensure that these folks understand the reality and importance of logging and monitoring to the overall information security effort.