Last Quick and Dirty Log Tip for the Week

OK, so this week I posted two other blog posts about doing quick and dirty log analysis and some of the techniques I use. This one also covers converting column logs to CSV.

After the great response, I wanted to drop one last tip for the week. 

Several folks asked me about re-sorting and processing the column-based data in different ways and to achieve different analytical views. 

Let me re-introduce you to my friend and yours, sort.

In this case, instead of using the sort -n -r like before (numeric sort, reverse order), we can use:

  • sort -k# -n input_file (where # is the number of the column you’d like to sort by and the input file is the name of the file to sort)
    • You can use this inline by leveraging the pipe (|) again – i.e.: cat input.txt | sort -k3 -n (this types the input file and sends it to sort for sorting on the third column in numeric order) (-r would of course, reverse it…)
    • You can write the output of this to a file with redirects “> filename.txt”, i.e.: cat input.txt | sort -k3 -n -r > output.txt
      • You could also use “>>” as the redirect in order to create a file if it doesn’t exist OR append to a file if it does exist… i.e..:  cat input.txt | sort -k3 -n -r >> appended_output.txt

That’s it! It’s been a fun week sharing some simple command line processing tips for log files. Drop me a line on Twitter (@lbhuston) and let me know what you used them for, or which ones are your favorite. As always, thanks and have a great weekend! 

Quick And Dirty Log Analysis Followup

Earlier this week, I posted some tips for doing Quick and Dirty PA Firewall Log Analysis.

After I posted this, I got a very common question, and I wanted to answer it here.

The question is something along the lines of “When I use the techniques from your post, the outputs of the commands are column separated data. I need them to be CSV to use with my (tool/SEIM/Aunt Gracie/whatever). How can I convert them?” Sound familiar?

OK, so how do we accomplish this feat of at the command line without all of the workarounds that people posted, and without EVER loading Excel? Thankfully we can use awk again for this.

We can use:

  • awk ‘BEGIN { OFS = “,”} ; {print $1,$2,$3}’
    • Basically, take an input of column data, and print out the columns we want (can be any, in this case I want the first 3 columns), and make the outputs comma delimited.
    • We can just append this to our other command stacks with another pipe (|) to get our output CSV
  • Example: cat log.csv | awk ‘BEGIN { FS = “,”} ; {print $8,$9}’ | sort -n | uniq -c | sort -n -r | awk ‘BEGIN { OFS = “,”} ; {print $1,$2,$3}’
    • In this example, the source IP and destination IP will be analyzed, and the reduced to unique pairs, along with the number of times that that pair is duplicated in the input log (I use this as a “hit rate” as I described earlier
      • A common question, why do I ask for two columns in the first awk and then ask for three columns in the second awk?
        • The answer of course, is that the first awk prints the unique pairs, but it also adds a column of the “hit rate”, so to get the output appropriately, I need all three fields.

So, once again, get to know awk. It is your friend.:)

PS – Yes, I know, there are hundreds of other ways to get this same data, in the same format, using other command line text processing tools. Many may even be less redundant than the commands above. BUT, this is how I did it. I think it makes it easy for people to get started and play with the data. Post your ways to Twitter or share with the community. Exploration is awesome, so it will encourage users to play more. Cool! Hit me on Twitter if you wanna share some or talk more about this approach (@lbhuston).

Thanks for reading!

How to Engage MSI for Supply Chain Security Help

The month of March is about to wrap up and come to a close. I hope it was a great month for you and your security initiatives. I also hope you took advantage of our focused content this month on Supply Chain Security. If you want to go back and read through some of the articles, here are quick links:

3 Reasons Your Supply Chain Security Stinks

Ideas for Vendor Discovery

Sorting Vendors into Tiers

Mapping Control Requirements to Vendor Tiers

An Example Control Matrix for Supply Chain Security

What is MSI’s Passive Assessment & How Does it Empower Supply Chain Security?

Many folks have asked us about how to engage with MSI around the Supply Chain. I wanted to add this bit of information in order to make it easier for folks to know how we can assist them.

You can engage with MSI around Supply Chain Security in three primary models:

  • Focused Mission Consulting Model – This model is when you have a specific set of tasks and deliverables in mind that you would like MSI to create/review/audit or test. We scope the work effort up front and provide a flat rate engagement price. The work is then completed, usually offsite, and the deliverables are worked through until completed. This is fantastic for organizations looking to build a program, create their tiers and control matrices and document the processes involved. Basically, you hire us to do the heavy lifting…
  • Retainer-Based Consulting Model – This model lets you hire MSI resources for a specific time frame (usually 1 year) for periodic oversight, design, review or operational tasks. Our team supplements your team, providing experience and assistance to your process. Basically, you do the heavy lifting – and we make sure you build an efficient, effective and safe program for supply chain security. This is a flat rate, billed monthly, for a set number of resource hours.
  • Virtual CISO Model – In this model, you can hire MSI to manage and provide oversight for security needs across the enterprise. You get an assigned MSI resource who is responsible for ensuring your initiatives get completed and performed in accordance with best practices. This resource can draw from other MSI subject matter experts and our services, as needed, to build out/supplement or support your security initiative. This is a great offering for small and mid-size organizations who need deep expertise, but who might not have the budget or capability to retain world class talent across multiple security domains. Basically, in this type of engagement – you hire us to solve your security problems and build/manage your security program. We do that with attention to cost/efficiency/effectiveness and safety. Pricing for this service type varies based on the maturity and requirements of the security program.

You can also retain MSI to leverage our passive assessment platform to assess your vendors passively, “en masse”. For information about how to engage with us to serve as a fulcrum for your security program, arrange for a free, no pressure, exploration call with your account executive. If you don’t have an account executive, give us a call at (614) 351-1237 or drop us a line at info (at) microsolved /dot/ com and let us know of your interest. We would love to share some demo information with you and walk you through how we can help.

If you have any other questions about Supply Chain Security or other issues, please get in touch, as above. You can also reach out to me on Twitter. As always, thanks for reading and until next time, stay safe out there!