Tag Archives: log analysis

FAQ on Audit Log Best Practices

Posted on by
Reply

Q: What are audit logs?

A: Audit logs are records of all events and security-related information that occur within a system. This information is crucial for incident response, threat detection, and compliance monitoring.

Q: Why is audit log management important?

A: Audit log management is essential for every organization that wants to ensure its data security. Without audit logs, organizations would have no way of knowing who accessed what information when or how the incident happened or whether unauthorized users or suspicious activity occurred. Moreover, audit log management supports compliance with industry regulations and guidelines.

Q: What are the best practices for audit log management?

A: To ensure that your audit log management practices meet the CIS CSC version 8 guidelines and safeguard requirements, consider implementing the following best practices:

1. Define the audit log requirements based on industry regulations, guidelines, and best practices.

2. Establish audit policies and procedures that align with your organization’s requirements and implement them consistently across all systems and devices.
3. Secure audit logs by collecting, storing, and protecting them securely to prevent unauthorized access or tampering.
4. Monitor and review audit logs regularly for anomalies, suspicious activity, and security violations, such as unauthorized access attempts, changes to access rights, and software installations.
5. Configure audit logging settings to generate records of critical security controls, including attempts to gain unauthorized access or make unauthorized changes to the network.
6. Generate alerts in real-time for critical events, including security violations, unauthorized access attempts, changes to access rights, and software installations.
7. Regularly test audit log management controls to ensure their effectiveness and meet your organization’s audit log requirements.

Q: What are the benefits of following audit log management best practices?

A: Following audit log management best practices can establish a strong framework for incident response, threat detection, and compliance monitoring. This, in turn, can help safeguard against unauthorized access, malicious activity, and other security breaches, prevent legal and financial penalties, and maintain trust levels with clients and partners.

Q: How long should audit logs be kept?

A: As a general rule, storage of audit logs should include 90 days hot (meaning actively available for immediate review or alerting), 6 months warm (meaning they can be restored within hours), and two years cold (meaning they can be restored within days). However, organizations should define retention periods based on their audit log requirements and compliance regulations. [1] [2]

*This article was written with the help of AI tools and Grammarly.

Let’s Talk About Audit Logs

Posted on by
Reply

CIS Control 8: Audit Log Management

Data is at the core of every business in today’s digital age. Protecting that data is of paramount importance. For this reason, the Center for Internet Security (CIS) developed the CIS Controls to provide a comprehensive framework for cybersecurity best practices.

One of these controls, CIS Control 8, focuses specifically on audit log management. This control aims to ensure that all events and security-related information are recorded and retained in an audit log for a defined period.

This article will explore the importance of audit log management as a fundamental component of any organization’s security posture. We will examine the CIS Control 8 safeguard requirements and industry-standard best practices for audit log management.

By following the procedures outlined in this article, organizations can improve their security posture, meet all CIS CSC version 8 safeguards, and ensure compliance with industry standards.

Why audit log management is essential

Audit log management is essential for every organization that wants to ensure its data security. The reason is simple: audit logs provide a comprehensive record of all events and security-related information that occurs within a system. This information is critical for incident response, threat detection, and compliance monitoring. Without audit logs, organizations would have no way of knowing who accessed what information, when or how the incident happened, or whether unauthorized users or suspicious activity occurred.

In addition to aiding in incident response and threat detection, audit log management also supports compliance with industry regulations and guidelines. Many compliance requirements mandate that organizations maintain a record of all activity that occurs on their systems. Failing to comply with these requirements can result in significant legal and financial penalties. Therefore, organizations prioritizing data security must take audit log management seriously and implement practices that meet their data security needs and safeguard requirements.

Best practices for audit log management

Audit log management is critical to an organization’s data security efforts. To ensure that your audit log management practices meet the CIS CSC version 8 guidelines and safeguard requirements, consider implementing the following best practices:

1. Define the audit log requirements: Assess the audit log requirements for your organization based on industry regulations, guidelines, and best practices. Define the data to be logged, audit events, and retention periods.

2. Establish audit policies and procedures: Develop audit policies and procedures that align with your organization’s requirements. Ensure these policies and procedures are implemented consistently across all systems and devices.

3. Secure audit logs: Audit logs should be collected, stored, and protected securely to prevent unauthorized access or tampering. Only authorized personnel should have access to audit logs.

4. Monitor and review audit logs: Regularly monitor and review audit logs for anomalies, suspicious activity, and security violations. This includes monitoring for unauthorized access attempts, changes to access rights, and software installations.

5. Configure audit logging settings: Ensure audit logs capture essential system information and user activity information. Configure audit logging settings to generate records of critical security controls, including attempts to gain unauthorized access or make unauthorized changes to the network.

6. Generate alerts: Configure the system to generate real-time alerts for critical events. This includes alerts for security violations, unauthorized access attempts, changes to access rights, and software installations.

7. Regularly test audit log management controls: Ensure audit log management controls are consistently implemented and reviewed. Conduct regular testing to ensure they are effective and meet your organization’s audit log requirements.

Organizations can establish a strong framework for incident response, threat detection, and compliance monitoring by implementing these best practices for audit log management. This will help safeguard against unauthorized access, malicious activity, and other security breaches, prevent legal and financial penalties, and maintain trust levels with clients and partners.

Audit log management policies

To establish audit log management policies that meet CIS CSC version 8 guidelines and safeguard requirements, organizations should follow the following sample policy:

1. Purpose: The purpose of this policy is to establish the principles for collecting, monitoring, and auditing all system and user activity logs to ensure compliance with industry regulations, guidelines, and best practices.

2. Scope: This policy applies to all employees, contractors, equipment, and facilities within the organization, including all workstations, servers, and network devices used in processing or storing sensitive or confidential information.

3. Policy:

– All computer systems and devices must generate audit logs that capture specified audit events, including user logins and accesses, system configuration changes, application accesses and modifications, and other system events necessary for detecting security violations, troubleshooting, and compliance monitoring.

– Audit logs must be generated in real-time and stored in a secure, centralized location that is inaccessible to unauthorized users.

– The retention period for audit logs must be at least 90 days, or longer if law or regulation requires.

– Only authorized personnel with appropriate access rights and clearances can view audit logs. Access to audit logs must be audited and reviewed regularly by the Information Security team.

– Audit logs must be reviewed regularly to identify patterns of suspicious activity, security violations, or potential security breaches. Any unauthorized access or security violation detected in the audit logs must be reported immediately to the Information Security team.

– Audit log management controls, and procedures must be tested periodically to ensure effectiveness and compliance with CIS CSC version 8 guidelines and safeguard requirements.

4. Enforcement: Failure to comply with this policy may result in disciplinary action, up to and including termination of employment or contract. All violations must be reported to the Information Security team immediately.

By implementing the above policy, organizations can ensure they meet the audit log management standards set forth by CIS CSC version 8 guidelines and safeguard requirements. This will help organizations prevent unauthorized access, malicious activity, and data breaches, maintain compliance with industry regulations, and protect the integrity and confidentiality of sensitive or confidential information.

Audit log management procedures

Here are the audit log management procedures that establish best practices for performing the work of this control:

I. Initial Setup

– Determine which audit events will be captured in the logs based on industry regulations, guidelines, and best practices.

– Configure all computer systems and devices to capture the specified audit events in the logs.

– Establish a secure, centralized location for storing the logs that is inaccessible to unauthorized users.

II. Ongoing Operations

– Set the logs to generate in real time.

– Monitor the logs regularly to detect security violations, troubleshoot, and monitor compliance.

– Ensure only authorized personnel with appropriate access rights can view the logs.

– Review the logs regularly to identify patterns of suspicious activity, security violations, or potential security breaches.

– Immediately report any unauthorized access or security violation detected in the logs to the Information Security team.

– Retain log data for at least 90 days, or longer if required by law or regulation.

III. Testing and Evaluation

– Test the audit log management controls and procedures periodically.

– Ensure that all testing and evaluation are conducted in compliance with CIS CSC version 8 guidelines and safeguard requirements.

By following these audit log management procedures, organizations can establish best practices for performing the work of this control and ensure that all system and user activities are properly monitored and audited. This will help organizations maintain compliance with industry regulations, prevent unauthorized access, and protect sensitive or confidential information from data breaches.

 

*This article was written with the help of AI tools and Grammarly.

Workstation Logging Best Practices

Posted on by
Reply

Why Workstation Logging Matters

Workstations are important components of any IT infrastructure, and they’re also one of the most overlooked. Often seen as expendable, many organizations fail to see the value of workstation logs, and how they can add to the visibility and detection capabilities of the security team. Workstations are quite likely to be early indicators of attack and malware infections. They are also often super useful in identifying manual attacker behaviors and performing adequate forensics.

Organizations that don’t maintain and organize workstation logs are usually missing out on some essential data and falling short of having across-the-enterprise visibility. This is especially true if you have a decentralized work environment. Simply enabling, configuring, and properly aggregating workstation logs can give you a huge forensic advantage. Adding real-time or near real-time log parsing and event alerting makes that advantage a superpower.

What to Log

The security events an organization captures on their workstations depend largely on industry-specific needs and relevant legal requirements. However, best practices call for several events that must be recorded and logged to ensure user accountability and to help organizations detect, understand, and recover from malicious events. These events include:

  • Authentication successes and failures for all users and services
  • Access control successes and failures for all users and services
  • Session activity, including files and applications used, especially system utilities and Powershell, if applicable
  • Changes in user access rights or privileges

The Bottom Line

Get busy logging on workstations. Make sure the logs are properly configured, aggregated, and processed as a part of your detection capabilities. Don’t view workstation logs as throw-aways. Instead, see them as a powerful lens for early detection, forensics, and attack recovery.

Update:

Thanks to @TheTokenFemale for pointing out that the logs should be sent somewhere off the system. I meant that by aggregation, but to clarify, the logs should be sent, processed, and archived using a log aggregation system or toolset that includes proper chain of evidence handling, alerting, and heuristics. It should also store and archive the relevant logs according to best practices and legal and regulatory guidance. 

What should be in a security log?

Posted on by
Reply

Logging is one of the most important aspects of any security program. It provides a record of events that occur within your environment, which allows you to understand how your systems are being used and what vulnerabilities exist. Logging helps you identify issues before they become problems, and it gives you insight into what happened after the fact.

There are many different types of logs, each with its own purpose. Some logs are designed to provide information about system activity, while others are intended to capture information about network traffic or application behavior. There are also different levels of logging, ranging from basic records of actions taken by applications, to detailed records of every event that occurs during the execution of an application.

In general, the more detail you can include in your logs, the better. For instance, if you’re looking for evidence of a compromise, you’ll need to look for signs of unauthorized access to your systems. A log entry that includes details about the IP addresses involved in the request will allow you to correlate the requests with the users making them. Similarly, if you’re trying to determine whether a particular file was accessed by someone else, you’ll need to examine the contents of the log entries associated with that file.

As you consider what type of logs to create, keep in mind that not all logs are created equal. In addition, not all logs are equally useful. For example, a log of HTTP requests might be helpful in determining whether a web server has been compromised, but it won’t tell you much about the nature of the threat. On the other hand, a log of failed login attempts could indicate that a malicious actor is attempting to gain access to your systems.

The best way to decide what kind of logs to create is to think about the specific threats you face and the kinds of information you want to collect. If you’re concerned about a particular type of threat, such as phishing emails, then you’ll probably want to track email messages sent to your domain. If you’re worried about malware infections, you’ll likely want to monitor the activities of your users’ computers.

In general, as a minimum, make sure the elements of the common logging format are included and build from there. If you need assistance with log design or help determining and implementing a logging strategy, drop us a line at info@microsolved.com. We’re happy to help! 

Last Quick and Dirty Log Tip for the Week

Posted on by
Reply

OK, so this week I posted two other blog posts about doing quick and dirty log analysis and some of the techniques I use. This one also covers converting column logs to CSV.

After the great response, I wanted to drop one last tip for the week. 

Several folks asked me about re-sorting and processing the column-based data in different ways and to achieve different analytical views. 

Let me re-introduce you to my friend and yours, sort.

In this case, instead of using the sort -n -r like before (numeric sort, reverse order), we can use:

  • sort -k# -n input_file (where # is the number of the column you’d like to sort by and the input file is the name of the file to sort)
    • You can use this inline by leveraging the pipe (|) again – i.e.: cat input.txt | sort -k3 -n (this types the input file and sends it to sort for sorting on the third column in numeric order) (-r would of course, reverse it…)
    • You can write the output of this to a file with redirects “> filename.txt”, i.e.: cat input.txt | sort -k3 -n -r > output.txt
      • You could also use “>>” as the redirect in order to create a file if it doesn’t exist OR append to a file if it does exist… i.e..:  cat input.txt | sort -k3 -n -r >> appended_output.txt

That’s it! It’s been a fun week sharing some simple command line processing tips for log files. Drop me a line on Twitter (@lbhuston) and let me know what you used them for, or which ones are your favorite. As always, thanks and have a great weekend! 

Quick And Dirty Log Analysis Followup

Posted on by
Reply

Earlier this week, I posted some tips for doing Quick and Dirty PA Firewall Log Analysis.

After I posted this, I got a very common question, and I wanted to answer it here.

The question is something along the lines of “When I use the techniques from your post, the outputs of the commands are column separated data. I need them to be CSV to use with my (tool/SEIM/Aunt Gracie/whatever). How can I convert them?” Sound familiar?

OK, so how do we accomplish this feat of at the command line without all of the workarounds that people posted, and without EVER loading Excel? Thankfully we can use awk again for this.

We can use:

  • awk ‘BEGIN { OFS = “,”} ; {print $1,$2,$3}’
    • Basically, take an input of column data, and print out the columns we want (can be any, in this case I want the first 3 columns), and make the outputs comma delimited.
    • We can just append this to our other command stacks with another pipe (|) to get our output CSV
  • Example: cat log.csv | awk ‘BEGIN { FS = “,”} ; {print $8,$9}’ | sort -n | uniq -c | sort -n -r | awk ‘BEGIN { OFS = “,”} ; {print $1,$2,$3}’
    • In this example, the source IP and destination IP will be analyzed, and the reduced to unique pairs, along with the number of times that that pair is duplicated in the input log (I use this as a “hit rate” as I described earlier
      • A common question, why do I ask for two columns in the first awk and then ask for three columns in the second awk?
        • The answer of course, is that the first awk prints the unique pairs, but it also adds a column of the “hit rate”, so to get the output appropriately, I need all three fields.

So, once again, get to know awk. It is your friend.:)

PS – Yes, I know, there are hundreds of other ways to get this same data, in the same format, using other command line text processing tools. Many may even be less redundant than the commands above. BUT, this is how I did it. I think it makes it easy for people to get started and play with the data. Post your ways to Twitter or share with the community. Exploration is awesome, so it will encourage users to play more. Cool! Hit me on Twitter if you wanna share some or talk more about this approach (@lbhuston).

Thanks for reading!

Quick & Dirty Palo Alto Log Analysis

Posted on by
Reply
OK, so I needed to do some quick and dirty traffic analysis on Palo Alto text logs for a project I was working on. The Palo Alto is great and their console tools are nice. Panorama is not too shabby. But, when I need quick and dirty analysis and want to play with data, I dig into the logs. 
 
That said, for my quick analysis, I needed to analyze a bunch of text logs and model the traffic flows. To do that, I used simple command line text processing in Unix (Mac OS, but with tweaks also works in Linux, etc.)
 
I am sharing some of my notes and some of the useful command lines to help others who might be facing a similar need.
 
First, for my project, I made use of the following field #’s in the text analysis, pulled from the log header for sequence:
  • $8 (source IP) 
  • $9 (dest IP)
  • $26 (dest port)
  • $15 (AppID)
  • $32 (bytes)
 
Once, I knew the fields that corresponded to values I wanted to study, I started using the core power of command line text processing. And in this case, the power I needed was:
  • cat
  • grep
    • Including, the ever useful grep -v (inverse grep, show me the lines that don’t match my pattern)
  • awk
    • particularly: awk ‘BEGIN { FS = “,”} ; {print $x, $y}’ which prints specific columns in CSV files 
  • sort
    • sort -n (numeric sort)
    • sort -r (reverse sort, descending)
  • uniq
    • uniq -c (count the numbers of duplicates, used for determining “hit rates” or frequency, etc.)
 
Of course, to learn more about these commands, simply man (command name) and read the details. 😃 
 
OK, so I will get you started, here are a few of the more useful command lines I used for my quick and dirty analysis:
  • cat log.csv | awk ‘BEGIN { FS = “,”} ; {print $8,$9,$26}’ | sort | uniq -c | sort -n -r > hitrate_by_rate.txt
    • this one produces a list of Source IP/Dest IP/Dest Port unique combinations, sorted in descending order by the number of times they appear in the log
  • cat log.csv | awk ‘BEGIN { FS = “,”} ; {print $8,$9}’ | sort -n | uniq -c | sort -n -r > uniqpairs_by_hitrate.txt
    • this one produces a list of the uniq Source & Destination IP addresses, in descending order by how many times they talk to each other in the log (note that their reversed pairings will be separate, if they are present – that is if A talks to B, there will be an entry for that, but if B initiates conversations with A, that will be a separate line in this data set)
  • cat log.csv | awk ‘BEGIN { FS = “,”} ; {print $15}’ | sort | uniq -c | sort -n -r > appID_by_hitrate.txt
    • this one uses the same exact techniques, but now we are looking at what applications have been identified by the firewall, in descending order by number of times that application identifier appears in the log
 
Again, these are simple examples, but you can tweak and expand as you need. This trivial approach to command line text analysis certainly helps with logs and traffic data. You can use those same commands to do a wondrous amount of textual analysis and processing. Learn them, live them, love them. 😃 
 
If you have questions, or want to share some of the ways you use those commands, please drop us a line on Twitter (@microsolved) or hit me up personally for other ideas (@lbhuston). As always, thanks for reading and stay safe out there! 

Daily Log Monitoring and Increased Third Party Security Responsibilities: Here They Come!

Posted on by
Reply

For years now we at MSI have extoled the security benefits of daily log monitoring and reciprocal security practices between primary and third party entities present on computer networks. It is constantly being proven true that security incidents could be prevented, or at least quickly detected, if system logs were properly monitored and interpreted. It is also true that many serious information security incidents are the result of cyber criminals compromising third party service provider systems to gain indirect access to private networks. 

I think that most large-network CISOs are well aware of these facts. So why aren’t these common security practices right now? The problem is that implementing effective log monitoring and third party security practices is plagued with difficulties. In fact, implementation has proven to be so difficult that organizations would rather suffer the security consequences than put these security controls in place. After all, it is cheaper and easier – usually – unless you are one of the companies that get pwned! Right now, organizations are gambling that they won’t be among the unfortunate – like Target. A fools’ paradise at best! 

But there are higher concerns in play here than mere money and efficiency. What really is at stake is the privacy and security of all the system users – which one way or another means each and every one of us. None of us likes to know our private financial or medical or personal information has been exposed to public scrutiny or compromise, not to mention identity theft and ruined credit ratings. And what about utilities and manufacturing concerns? Failure to implement the best security measures among power concerns, for example, can easily lead to real disasters and even loss of human life. Which all means that it behooves us to implement controls like effective monitoring and vendor security management. There is no doubt about it. Sooner or later we are going to have to bite the bullet. 

Unfortunately, private concerns are not going to change without prodding. That is where private and governmental regulatory bodies are going to come into play. They are going to have to force us to implement better information security. And it looks like one of the first steps in this process is being taken by the PCI Security Standards Council. Topics for their special interest group projects in 2015 are going to be daily log monitoring and shared security responsibilities for third party service providers.

That means that all those organizations out there that foster the use of or process credit cards are going to see new requirements in these fields in the next couple of years. Undoubtedly similar requirements for increased security measures will be seen in the governmental levels as well. So why wait until the last minute? If you start now implementing not only effective monitoring and 3rd party security, but other “best practices” security measures, it will be much less painful and more cost effective for you. You will also be helping us all by coming up with new ways to practically and effectively detect security incidents through system monitoring. How about increasing the use of low noise anomaly detectors such as honey pots? What about concentrating more on monitoring information leaving the network than what comes in? How about breaking massive networks into smaller parts that are easier monitor and secure? What ideas can you come up with to explore?

This post written by John Davis.

Remember, Log Analysis is Important, Especially Now

Posted on by
Reply

Remember, during the holiday season, attacks tend to increase and so do compromises. With vacations and staff parties, monitoring the logs and investigating anomalies can quickly get forgotten. Please make sure you remain vigilant during this time and pay close attention to logs during and just after holiday breaks.

As always, thanks for reading and we wish you a safe and happy holiday season!

Monitoring: an Absolute Necessity (but a Dirty Word Nonetheless)

Posted on by
3

There is no easier way to shut down the interest of a network security or IT administrator than to say the word “monitoring”. You can just mention the word and their faces fall as if a rancid odor had suddenly entered the room! And I can’t say that I blame them. Most organizations do not recognize the true necessity of monitoring, and so do not provide proper budgeting and staffing for the function. As a result, already fully tasked (and often times inadequately prepared) IT or security personnel are tasked with the job. This not only leads to resentment, but also virtually guarantees that the job is will not be performed effectively.

And when I say human monitoring is necessary if you want to achieve any type of real information security, I mean it is NECESSARY! You can have network security appliances, third party firewall monitoring, anti-virus packages, email security software, and a host of other network security mechanisms in place and it will all be for naught if real (and properly trained) human beings are not monitoring the output. Why waste all the time, money and effort you have put into your information security program by not going that last step? It’s like building a high and impenetrable wall around a fortress but leaving the last ten percent of it unbuilt because it was just too much trouble! Here are a few tips for effective security monitoring:

  • Properly illustrate the necessity for human monitoring to management, business and IT personnel; make them understand the urgency of the need. Make a logical case for the function. Tell them real-world stories about other organizations that have failed to monitor and the consequences that they suffered as a result. If you can’t accomplish this step, the rest will never fall in line.
  • Ensure that personnel assigned to monitoring tasks of all kinds are properly trained in the function; make sure they know what to look for and how to deal with what they find.
  • Automate the logging and monitoring function as much as possible. The process is difficult enough without having to perform tedious tasks that a machine or application can easily do.
  • Ensure that you have log aggregation in place, and also ensure that other network security tool output is centralized and combined with logging data. Real world cyber-attacks are often very hard to spot. Correlating events from different tools and processes can make these attacks much more apparent. 
  • Ensure that all personnel associated with information security communicate with each other. It’s difficult to effectively detect and stop attacks if the right hand doesn’t know what the left hand is doing.
  • Ensure that logging is turned on for everything on the network that is capable of it. Attacks often start on client side machines.
  • Don’t just monitor technical outputs from machines and programs, monitor access rights and the overall security program as well:
  • Monitor access accounts of all kinds on a regular basis (at least every 90 days is recommended). Ensure that user accounts are current and that users are only allocated access rights on the system that they need to perform their jobs. Ensure that you monitor third party access to the system to this same level.
  • Pay special attention to administrative level accounts. Restrict administrative access to as few personnel as possible. Configure the system to notify proper security and IT personnel when a new administrative account is added to the network. This could be a sign that a hack is in progress.
  • Regularly monitor policies and procedures to ensure that they are effective and meet the security goals of the organization. This should be a regular part of business continuity testing and review.
Thanks to John Davis for writing this post.