Basic Logging Advice

Logging and monitoring are two important aspects of any security program. Without logging, we cannot understand how our systems operate, and without monitoring, we cannot detect anomalies and issues before they become problems.

There are many different types of logs available to us today. Some are generated automatically, while others require manual intervention. For instance, network traffic is usually logged automatically. However, application logs are not. We may need to manually create these logs.

Application logs provide valuable information about what happened during the execution of an application. They can show us which parts of the application were executed, what resources were used, and what was returned. Application logs are often stored in databases, allowing us to query them later.

Network logs are also useful. They allow us to see what packets were sent and received, and what responses were made. 

System logs are another type of log that we should consider. System logs record events such as system startup, shutdown, reboots, etc. They are generally stored in files, but can also be recorded in databases.

While logs are very helpful, they do have their limitations:

  • First, logs are only as good as the people who generate them. If  something doesn’t save a log, then we likely don’t know what happened. We might be able to get that from some other log, but having multiple layers of logs around an event is often useful.
  • Second, logs are static. Once created, they should remain unchanged. Hashing logs, storing them on read only file systems and other forms of log controls are highly suggested.
  • Third, logs are not always accurate. Sometimes, logs contain false positives, meaning that something appears to be happening when actually nothing is. False negatives are also possible, meaning we don’t alert on something we should have. Logs are a part of detection solution, not the sole basis of one.
  • Fourth, logs are not always actionable. That means that we can’t easily tell from a log whether something bad has occurred or if it is just noise. This is where log familiarity and anomaly detection comes in. Sometimes reviewing logs in aggregate and looking for trends is more helpful than individual line by line analysis. The answer may be in looking for haystacks instead of needles…
  • Finally, logs are not always timely. They might be created after the fact, and therefore won’t help us identify a problem until much later. While good log analysis can help create proactive security through threat intelligence, they are more powerful when analyzing events that have happened or as sources for forensic data.

Keep all of these things in mind when considering logging tools, designing monitoring techniques or building logs for your systems and applications.

How often should security logs be reviewed?

Security logs are one of the most important components of any security program. They provide insight into how well your security program is working, and they serve as a valuable source of intelligence for incident response. However, they are not perfect; they can contain false positives and false negatives. As a result, they need to be reviewed regularly to ensure they are providing accurate information.

There are two main reasons why security log reviews are necessary. First, they allow you to identify problems before they become serious incidents. Second, they allow you to determine whether your current security measures are effective.

When reviewing logs, look for three things:

1. Incidents – These are events that indicate something has gone wrong. For example, a firewall blocking access to a website, or a virus scanning software alerting you to a malware infection.

2. False Positives – These are alerts that don’t represent anything actually happening. For example, a virus scanner warning you about a file that was downloaded from the Internet without any infection identified.

3. False Negatives – These are alerts that do represent something actually happening, but were missed because of a flaw in the system. For example, a server being accessed remotely, but no alarms raised.

Reviewing logs every day is recommended. If you review logs daily, you will catch issues sooner and prevent them from becoming major incidents. This should be done on a rotating basis by the security team to prevent fatigue from diminishing the quality of the work, or via automated methods to reduce fatigue.

Peer reviewing logs weekly is also recommended. It allows you to spot trends and anomalies that might otherwise go unnoticed by a single reviewer. It also gives a second set of eyes on the logs, and helps guard against fatigue or bias-based errors.

Finally, aggregated trend-based monthly reviews are recommended. This gives you a chance to look back and see if there have been any changes to your environment that could affect your security posture or represent anomalies. This is a good place to review items like logged events per day, per system, trends on specific log events and the like. Anomalies should be investigated. Often times, this level of log review is great for spotting changes to the environment or threat intelligence.

If you want to learn more about how to conduct log reviews effectively, reach out to us at info@microsolved.com. We’re happy to help!

How long should security logs be kept?

Security logs are a great source of information for incident response, forensics, and compliance purposes. However, log retention policies vary widely among organizations. Some keep logs indefinitely; others only retain them for a certain period of time. Logging practices can impact how much useful information is available after a compromise has occurred.

In general, the longer logs are retained, the better. But, there are several factors to consider when determining how long to keep logs. These include:

• What type of system is being monitored?

• Is the system mission-critical?

• Are there any legal requirements regarding retention of logs?

• Does the company have a policy regarding retention of logs? If so, does it match industry standards?

• How often do incidents occur?

• How many employees are affected by each incident?

• How many incidents are reported?

• How many hours per day are logs collected?

• How many days per week are logs collected?

It is important to understand the business needs before deciding on a retention policy. For example, if a company has a policy of retaining logs for 90 days, then it is reasonable to assume that 90 days is sufficient for the majority of situations. However, if a company has no retention policy, then it is possible that the logs could be lost forever.

Logs are one of the most valuable sources of information during an investigation. It is important to ensure that the right people have access to the logs and that they are stored securely. In addition, it is important to know how long logs need to be kept.

MicroSolved provides a number of services related to logging and monitoring. We can help you create logging policies and practices, as well as design log monitoring solutions. Drop us a line at info@microsolved.com if you’d like to discuss logging and logging solutions.

What should be in a security log?

Logging is one of the most important aspects of any security program. It provides a record of events that occur within your environment, which allows you to understand how your systems are being used and what vulnerabilities exist. Logging helps you identify issues before they become problems, and it gives you insight into what happened after the fact.

There are many different types of logs, each with its own purpose. Some logs are designed to provide information about system activity, while others are intended to capture information about network traffic or application behavior. There are also different levels of logging, ranging from basic records of actions taken by applications, to detailed records of every event that occurs during the execution of an application.

In general, the more detail you can include in your logs, the better. For instance, if you’re looking for evidence of a compromise, you’ll need to look for signs of unauthorized access to your systems. A log entry that includes details about the IP addresses involved in the request will allow you to correlate the requests with the users making them. Similarly, if you’re trying to determine whether a particular file was accessed by someone else, you’ll need to examine the contents of the log entries associated with that file.

As you consider what type of logs to create, keep in mind that not all logs are created equal. In addition, not all logs are equally useful. For example, a log of HTTP requests might be helpful in determining whether a web server has been compromised, but it won’t tell you much about the nature of the threat. On the other hand, a log of failed login attempts could indicate that a malicious actor is attempting to gain access to your systems.

The best way to decide what kind of logs to create is to think about the specific threats you face and the kinds of information you want to collect. If you’re concerned about a particular type of threat, such as phishing emails, then you’ll probably want to track email messages sent to your domain. If you’re worried about malware infections, you’ll likely want to monitor the activities of your users’ computers.

In general, as a minimum, make sure the elements of the common logging format are included and build from there. If you need assistance with log design or help determining and implementing a logging strategy, drop us a line at info@microsolved.com. We’re happy to help! 

Monitoring: an Absolute Necessity (but a Dirty Word Nonetheless)

There is no easier way to shut down the interest of a network security or IT administrator than to say the word “monitoring”. You can just mention the word and their faces fall as if a rancid odor had suddenly entered the room! And I can’t say that I blame them. Most organizations do not recognize the true necessity of monitoring, and so do not provide proper budgeting and staffing for the function. As a result, already fully tasked (and often times inadequately prepared) IT or security personnel are tasked with the job. This not only leads to resentment, but also virtually guarantees that the job is will not be performed effectively.

And when I say human monitoring is necessary if you want to achieve any type of real information security, I mean it is NECESSARY! You can have network security appliances, third party firewall monitoring, anti-virus packages, email security software, and a host of other network security mechanisms in place and it will all be for naught if real (and properly trained) human beings are not monitoring the output. Why waste all the time, money and effort you have put into your information security program by not going that last step? It’s like building a high and impenetrable wall around a fortress but leaving the last ten percent of it unbuilt because it was just too much trouble! Here are a few tips for effective security monitoring:

  • Properly illustrate the necessity for human monitoring to management, business and IT personnel; make them understand the urgency of the need. Make a logical case for the function. Tell them real-world stories about other organizations that have failed to monitor and the consequences that they suffered as a result. If you can’t accomplish this step, the rest will never fall in line.
  • Ensure that personnel assigned to monitoring tasks of all kinds are properly trained in the function; make sure they know what to look for and how to deal with what they find.
  • Automate the logging and monitoring function as much as possible. The process is difficult enough without having to perform tedious tasks that a machine or application can easily do.
  • Ensure that you have log aggregation in place, and also ensure that other network security tool output is centralized and combined with logging data. Real world cyber-attacks are often very hard to spot. Correlating events from different tools and processes can make these attacks much more apparent. 
  • Ensure that all personnel associated with information security communicate with each other. It’s difficult to effectively detect and stop attacks if the right hand doesn’t know what the left hand is doing.
  • Ensure that logging is turned on for everything on the network that is capable of it. Attacks often start on client side machines.
  • Don’t just monitor technical outputs from machines and programs, monitor access rights and the overall security program as well:
  • Monitor access accounts of all kinds on a regular basis (at least every 90 days is recommended). Ensure that user accounts are current and that users are only allocated access rights on the system that they need to perform their jobs. Ensure that you monitor third party access to the system to this same level.
  • Pay special attention to administrative level accounts. Restrict administrative access to as few personnel as possible. Configure the system to notify proper security and IT personnel when a new administrative account is added to the network. This could be a sign that a hack is in progress.
  • Regularly monitor policies and procedures to ensure that they are effective and meet the security goals of the organization. This should be a regular part of business continuity testing and review.
Thanks to John Davis for writing this post.

Splunk 4 Review

For this weeks tool review, we’re looking at Splunk. Splunk is a log collection engine at heart, but it’s really more than that. Think of it as search engine for your IT infrastructure. Splunk will actually collect and index anything you can throw at it, and this is what made me want to explore it.

Setting up your Splunk server is easy, there’s installers for every major OS. Run the installer and visit the web front end, and you are in business. Set up any collection sources you need, I started off with syslog. I started a listener in Splunk, and then forwarded my sources to Splunk (I used syslog-ng for this). Splunk will also easily do WMI polling, monitoring local files, change monitoring, or run scripts to generate any data you want. Some data sources require running Splunk as an agent, but it goes easy on system resources as the GUI is turned off. Installing agents is exactly the same process — you just disable the GUI when you’re finished setting up; however you can still control Splunk through the command line.

Splunk can also run addons, in the form of apps. These are plugins that are designed to take and display certain information. There are quite a few, provided both by the Splunk team and also some created by third parties. I found the system monitoring tools to be very helpful. There are scripts for both Windows and Unix. In this instance, it does require running clients on the system. There are also apps designed for Blue Coat, Cisco Security and more.

In my time using Splunk, I’ve found it to be a great tool for watching logs for security issues (brute forcing ssh accounts for example), it was also useful in fine tuning my egress filtering, as I could instantly see what was being blocked by the firewall, and of course the system monitoring aspects are useful. It could find a home in any organization, and it plays nice with other tools or could happily be your main log aggregation system.

Splunk comes in two flavors, free and professional. There’s not a great difference between them. The biggest difference is that with the free version Splunk is limited to 500MB of indexing per day, which proves to be more than enough for most small businesses, and testing for larger environments. Stepping up to the professional version is a lot easier on the pockets than might be expected, only about $3,000.