Q: “I have massive amounts of log files I have to dig through every day. I have tried a full blown SEIM, but can’t get it to work right or my management to support it with budget. Right now I have Windows logs, firewall logs and AV logs going to a syslog server. That gives me a huge set of text files every day. How can I make sense of all that text? What tools and processes do you suggest? What should I be looking for? HELP!!!!”

Adam Hostetler answered with:

I would say give OSSEC a try. It’s a free log analyzer/SEIM. It doesn’t

have a GUI with100 different dashboards and graphs, it’s all cli and

e-mail based (though there is a simple web interface for it also). It is

easy to write rules for, and it has default rules for many things,

except for your AV. You can write simple rules for that, especially if

you are just looking for items AV caught. It does take some tuning, as

with all analysis tools, but isn’t difficult after learning how OSSEC

works. If you want to step it up a bit, you can feed OSSEC alerts into

Splunk where you can trend alerts, or create other rules and reports in it.

Bill Hagestad added:

First things first – don’t be or feel overwhelmed – log files are what they are much disparate data from a variety of resources that need reviewing sooner rather than later.

Rather than looking at another new set to tools or the latest software gizmo the trade rags might suggest based on the flair of the month, try a much different and more effective approach to the potential threat surface to your network and enterprise information network.

First take a look at what resources need to be protected in order of importance to your business. Once you have prioritized these assets then begin to  determine what is the minimum level of acceptable risk you can assign to each resource you have just prioritized.

Next, make two columns on a either a piece of paper or a white board. In one column list your resources in order of protection requirements, i.e.; servers with customer data, servers with intellectual property, so and so forth. In a column to the right of the first assets list plug in your varying assigned levels of risk. Soon you will see what areas/assets within your organization/enterprise you should pay the most attention to in terms of threat mitigation.

After you have taken the steps to determine your own self- assessment of risk contact MicroSolved for both a vulnerability assessment and penetration test to provide additional objective perspective on threats to your IT infrastructure and commercial enterprise. 

Finally, Jim Klun weighed in with: 

You are way ahead of the game by just having a central log repository.  You can go to one server and look back in time to the point where you expect a security incident.

And what you have – Windows logs, firewall logs, and AV – is fantastic.  Make sure all your apps are logging as well ( logon success, logon failure).

Too often I have seen apps attacked and all I had in syslog was OS events that showed nothing.

Adam’s suggestion, OSSEC, is the way to go to keep cost down… but don’t just install and hope for the best.

You will have to tweak the OSSEC rules and come up with what works.

Here’s the rub: there is no substitute for knowing your logs – in their raw format, not pre-digested by a commercial SIEM or OSSEC.

That can seem overwhelming. And to that, some Unix commands and regular expressions are your friend.

So:

zcat auth.log | grep ssh | egrep -i ‘failed|accepted’

produces:

Jul  4 16:32:16 dmz-server01 sshd[8786]: Failed password for user02 from 192.168.105.51 port 38143 ssh2

Jul  4 16:33:53 dmz-server01 sshd[8786]: Accepted password for user01 from 192.168.105.38 port 38143 ssh2

Jul  4 16:36:05 dmz-server01 sshd[9010]: Accepted password for user01 from 192.168.105.38 port 38315 ssh2

Jul  5 01:04:00 dmz-server01 sshd[9308]: Accepted password for user01 from 192.168.105.38 port 60351 ssh2

Jul  5 08:21:58 dmz-server01 sshd[9802]: Accepted password for user01 from 192.168.105.38 port 51436 ssh2

Jul  6 10:21:52 dmz-server01 sshd[21912]: Accepted password for user01 from 192.168.105.38 port 36486 ssh2

Jul  6 13:43:10 dmz-server01 sshd[31701]: Accepted password for user01 from 192.168.105.30 port 34703 ssh2

Jun 26 11:21:02 dmz-server01 sshd[31950]: Accepted password for user01 from 192.168.105.70 port 37209 ssh2

Instead of miles of gibberish the log gets reduced to passed/fail authentication attempts.

You can spend an hour with each log source ( firewall, AV, etc) and quickly pare them down to whats interesting.

Then make SURE your OSSEC  rules cover what you want to see.

If that does not work – cron a script to parse the logs of interest using your regular expression expertise and have an email sent to you when something goes awry.

Revisist the logs manually periodically – they will change. New stuff will happen.  Only a human can catch that.

Take a look at:

http://www.securitywarriorconsulting.com/logtools/

The site lists a number of tools that may be useful

John Davis added:

You voice one of the biggest problems we see in information security programs: monitoring! People tell us that they don’t have the proper tools and, especially, they don’t have the manpower to perform effective logging and monitoring. And what they are saying is true, but unfortunately doesn’t let them out from having to do it. If you have peoples financial data, health data (HIPAA) or credit card information (PCI) you are bound by regulation or mandate to properly monitor your environment – and that means management processes, equipment, vulnerabilities and software as well as logs and tool outputs. The basic problem here is that most organizations don’t have any dedicated information security personnel at all, or the team they have isn’t adequate for the work load. Money is tight and employees are expensive so it is very difficult for senior management to justify the expenditure – paying a third party to monitor firewall logs is cheaper. But for real security there is no substitute for actual humans in the security loop – they simply cannot be replaced by technology. Unfortunately, I feel the only answer to your problem is for government and industry to realize this truth and mandate dedicated security personnel in organizations that process protected data.

As always, thanks for reading and if you have a question for the experts, either leave it in the comments, email us or drop us a line on Twitter at (@lbhuston).