Fuzzing Optical Smart Meters with ProtoPredator

PPClawsWords1

Our team has been working hard in the lab, once again testing the optical implementations of a variety of smart meters. Using our proprietary in-house developed tool, called ProtoPredator for Smart Meters, we have been doing full fuzzing of optical protocol implementations. 

Our tool makes this process easy and reproducible. It also provides for easy regression testing and fix validation through session replays. 

One of the things that makes ProtoPredator so cool is that it includes both arbitrary conversations with the meters in addition to canned sessions, making much more flexible in the hands of a knowledgeable user. You can easily use this feature to perform more nuanced validation of the protocols, testing things like sequence errors, poor trust, error recovery, etc. 

While ProtoPredator is still tied to the optical coupler speed and the inherent speed of the protocols in use, testing with it makes validation of the optical ports more effective than other more traditional approaches. Additionally, you can use multiple seats of ProtoPredator in parallel to decrease the overall testing and validation time, especially since the “brain files” and packet sessions are easily interchangeable amongst installations.

The easy to use GUI also means less frustration and more time on task for most users. It lets the testers spend less time on mundane tasks like serial configuration and hand crafting packets and more time on security testing, protocol analysis and bug hunting.

To find out more about ProtoPredator, or to discuss having our lab give your smart meters a look over, get in touch. Info(at)micro solved(dot)com will get you a prompt response. As always, thanks for reading! 

3 Tough Questions with Chris Jager

Recently, I got to spend some time interviewing Chris Jager via email on industrial control systems security. He didn’t pull any punches and neither did I. Here, are 3 Tough Questions between myself (@lbhuston) and Chris.


A Short Biography of Chris Jager (@chrisjager): I have over 15 years of experience in Information Technology and have focused on the practical application of security principles throughout my career. Most recently, I was director of the NESCO Tactical Analysis Center at EnergySec; a non-profit organization formed to facilitate information sharing, situational awareness, and education outreach to the energy sector. I am active in a number of information security workgroups and have provided operational, architectural, and regulatory compliance guidance to large and small organizations in both the public and private sectors, focusing on the energy sector exclusively since 2006.


Brent: You have spent a lot of time working on Industrial Control Systems (ICS) in your career. During that time, you have been witness to the explosion of interest in IT security as a profession. Why should some of the younger folks thinking about information security as a career consider a focus on ICS and SCADA? Why should they care?

Mr. Jager: This is a fantastic question and, if I frame my response correctly, the answer will hopefully be self-evident to your readers.

ICS and SCADA are terms that are seldom understood and often misused by information security (infosec) publications. SCADA systems typically manage geographically disperse areas and often consist of numerous functionally disparate processes.

However, because of the immense variety of different processes that can be managed by industrial control systems, ICS has become somewhat of a catchall term – including SCADA systems. For example, you’ll often find electric power generation processes such as turbine control, burner management, vibration monitoring and more lumped into the mix. Each of these processes has discrete or locally distributed control and instrumentation systems, any of which can cause catastrophic safety, reliability, and financial issues if misused.

For me, the challenge of protecting these kinds of systems is far more interesting than making sure that little Bobby can’t drop the student records table in a classroom database. Much of the actual management technology is the same as what is used in general IT, but the application is very different. Things get a little more exotic (and arcane) when you go further down the stack into digital–to-analog conversion, but it’s not overly difficult for most folks to understand once exposed to it. The negative impacts of misuse aren’t limited to convenience and financial loss. Risk to life and limb is a very real possibility in many processes that are managed by industrial control system automation that is being run out of specification.

Typically, industrial control systems are deployed in step with the physical equipment they are designed to manage. The physical equipment is often orders of magnitude more expensive than the ICS components that ship with it and may be designed for lifespans measured in decades. In short, upgrades seldom occur as they need to be engineered and tested for functionality, safety, and a myriad of other issues pertaining to the existing physical equipment.

This has led to a situation where the groups that understand control systems and processes are naturally (and often generationally) gapped from those groups who understand the current threat and vulnerability landscapes. Consequently, there are currently very few individuals that understand industrial control system security as it relates to the changing threat picture. If the challenge of doing something very few dare to try doesn’t sound good on its own, this is the sound of opportunity knocking. Answer the door!

I’d like to make one last point on this question. Take a look around your house or apartment and count the number of internet-enabled devices you have. Most people these days have far fewer traditional computers than embedded systems – devices that aren’t user-serviceable without breaking a warranty or two. And the hacking skills necessary to modify such devices to fit use cases unintended by the manufacturers seem to come naturally to the younger folk of today. Those skills are also relatively portable to the ICS/SCADA world where embedded systems are the norm. Sure, some of the protocols and hardware packages are somewhat different, but they are all relatively simple compared to what folks are tinkering with at their coffee tables. We can always use more QA/breakers – particularly at key points in the supply chain where issues can be spotted and fixed before they become permanently vulnerable installations. Again I say, “knock knock”!

 

Brent: You talk a lot about how challenging ICS/SCADA security is. Do you think protecting ICS/SCADA systems in a meaningful way is an attainable goal? Assuming yes, do you think it could be done during what’s left of our careers? Why or Why not?

Mr. Jager: If I didn’t think it was an attainable goal, I’d not be doing the kind of work I’ve done over the past number of years. There are much easier ways to make a buck than to have people who are entrenched in the old way of doing things actively work to prevent you from even introducing discussions about change – let alone actually implementing it!

There is momentum in this area, but much work still needs to be done. Devices still ship from manufacturers with easily discerned hardcoded administration credentials, firmware updates are accepted without challenge and more. Once deployed in the field, user passwords seldom change, vulnerabilities discovered post-installation go unmitigated, and so on.

Because we have all this noise around basic security failures and their associated issues, we don’t yet know what constitutes “meaningful” or “attainable” when we speak of complex industrial control systems. A prime example here is that the electric sector is still using the exact same set of controls and asset scoping for its regulated security standards as when I first started working in the sector in 2006. NERC CIP version 1 was in final draft form, and the current requirements catalog will remain largely unchanged until at least 2015 when and if version 5 becomes effective. There have been minor changes in the interim, but not one that comes remotely close to addressing change in the threat landscape.

Will we ever have a perfect system? No. We do, however, urgently need to stop being complacent about the subject and implement those security measures that we can.

 

Brent: If you had your own ICS system, let’s say you ran Chris’s power company, what would that look like? How would it be protected?

Mr. Jager: It would look very, very “dumb”. Until such time as ICS and other automation technologies are vetted by process engineers – and I’m talking about the entire ICS/automation stack, I would automate only where it was impossible to operate the business or process without it.

It seems to me that we have a major employment problem in this country and no clear path to resolution. Putting some of these people to work securing our industrial control systems is an area where the private sector can help get the country back to work without relying on government funded stimulus packages. An added bonus is that we’ll end up with a whole cadre of workers who have been exposed to the industry, a percentage of who will stay in the field and help to address the industry’s gray out problem. All it takes is one or two sizable impacts from automation failure or misuse for the cost savings seen through automation to be wiped out.

Where I had no choice but to automate, Chris’ Power Company would look very much like any power company out there today, unfortunately. There simply aren’t enough vendors and manufacturers out there presently that produce secure equipment. Even then, systems integrators often further weaken the environment by adding support accounts and other remotely accessible backdoors to these systems.

Be it in the energy sector or any other, process automation installations will inevitably mature to a state of persistent vulnerability due to their long lifespans. Vulnerability discovery and exploitation techniques advance over time, vulnerabilities are introduced through regression bugs elsewhere in the software or protocol stack, or the process itself may have changed to a point where a previously innocuous vulnerability now has the ability to introduce a large impact if exploited.

Eventually, pointing out that the emperor has no clothes becomes a career limiting move – particularly when said emperor is an exhibitionist! Instead, the focus should be on identifying the more sensitive individuals in the crowd and protecting them appropriately through sound risk identification principles. We can’t make the problems go away through risk management, but we can use the techniques to identify the things that matter most and, where we can’t mitigate the risk, implement monitoring and response controls. This sort of approach also helps prioritize future efforts and dollars.

The top security controls at Chris’ Power Company would center around monitoring and response as employees would be trained to assume the environment was in a persistent state of compromise. In the environment we live in today where threats are real and expressed, and vulnerabilities aren’t able to be universally mitigated, the only real chance at controlling risk you have is to manage the impact of a successful attack. You only get that chance if you are able to detect and respond before the attack balloons to the maximum impact value.

If you failed to give my company that chance, you wouldn’t be working at Chris’ Power Company!


Thanks to Chris Jager for his insights and passion about ICS security. We appreciate his willingness to spend time with us. Thanks, as always, to you the reader, for your attention. Until next time, stay safe out there!

Event Announcement: ICS/SCADA Security Briefing

MSI, along with the teams at NexDefense and Critical Intelligence, will be participating in an online webinar about ICS/SCADA Security. The date of the event is February, 6th and you can learn more about it here

The event is free to attend, though registration is required. You can earn a CPE for participating! 

We hope you will tune in and check us out!

Overview of the event: 

Learning Objectives

  • Significant trends in the threat and vulnerability environment
  • Relevant trends in ICS technology
  • What proactive steps you can take
  • How to leverage security intelligence

Agenda

  • Introductions
  • ICS Cyber Security Intelligence Briefing, Michael Assante
  • ICS Threat Update, Brent Huston
  • How to Leverage Security Intelligence, Bob Huber
  • Live Q&A

Who Should View?

  • Senior Information Security Leaders, CISOs and CTOs
  • Security and Risk Analysts
  • Control system security engineers
  • Security operation leads for ICS reliant organizations

SANS SCADA Security Conference & a DISCOUNT

SANS has allowed us to offer a 10% discount to our readers who attend their SCADA Security Summit. The event is being held in Orlando this year, February 12-13, with optional training courses wrapped around on both sides. We think this is a great event and we are proud to be able to help SANS promote it.

You can get your discount using the discount code: MicroSolvedSCADA

More information about the event follows below (Overview provided by SANS): 

More than 1,200 security analysts and process control engineers, from government and industry, have attended the SCADA Security Summits. That’s because Summits are the one place where the people shaping the future of control systems security come together to share the lessons they have learned and because the Summits give attendees unique, early access to important new information. This year’s program will be no different. If you have any responsibility for security of control systems – policy, engineering, governance or operations you won’t want to miss the 2013 Summit in Orlando, Florida.

 At the Summit you will:

  • Learn why control systems are so difficult to protect and arm yourself with clear case studies showing what’s been done and what can be done to protect SCADA and other control systems.
  • Learn the language of control systems so you can be of more help to the engineers who plan and deploy such systems.
  • Understand the requirements and constraints faced by owners and operators of automation systems. Determine the state of the art in control system security as a benchmark for your own future planning.
  • How to build an ICS security program and develop your team.
  • Better understand what government can and can’t do by learning the requirements, constraints and current capabilities available to secure critical control systems.

 For more information and to register click here  http://www.sans.org/event/north-american-scada-2013

Threat Data Sharing in ICS/SCADA Needs Improvement

I had an interesting discussion on Twitter with a good friend earlier this week. The discussion was centered around information sharing in ICS/SCADA environments – particularly around the sharing of threat/attack pattern/vulnerability data. 

It seems to us that this sharing of information – some might call it “intelligence”, needs to improve. My friend argues that regulation from the feds and local governments have effectively made utilities and asset owners so focused on compliance, that they can’t spare the resources to share security information. Further, my friend claims that sharing information is seen as dangerous to the utility, as if the regulators ever found out that information was shared that wasn’t properly reported “up the chain”, that it could be used against the utility to indicate “negligence” or the like. I can see some of this, and I remember back to my DOE days when I heard some folks talk along the same lines back when we showed up to audit their environments, help them with incidents or otherwise contribute to their information security improvement.

When I asked on open Twitter with the #ICS/#SCADA hashtags about what hampered utilities from sharing information, the kind Twitter folks who replied talked about primarily three big issues: the lack of a common language for expressing security information (we have some common languages for this (mitre’s work, VERIS, etc.)), legal/regulatory concerns (as above) and the perceived lack of mitigations available (I wonder if this is apathy, despair or a combination of both?). 

I would like to get some wider feedback on these issues. If you don’t mind, please let me know either in comments, via private email or via Twitter (@lbhuston) what you believe the roadblocks are to information sharing in the ICS/SCADA community.

Personally, I see this as an area where a growth of “community” itself can help. Maybe if we can build stronger social ties amongst utilities, encourage friendship and sharing at a social level, empower ourselves with new mechanisms to openly share data (perhaps anonymously) and create an air of trust and equity, we can solve this problem ourselves. I know the government and industry has funded ISACs and other organizations, but it seems to me that we need something else – something more easily participatory, more social. It has to be easier and safer to share information between us than it is today. Maybe, if we made such a thing, we could all share more openly. That’s just my initial 2 cents. Please, share yours.

Thanks for reading, and until next time, stay safe out there!  

ProtoPredator to Become Family of Products

On Nov 16, we announced the availability of ProtoPredator for Smart Meters (PP4SM). That tool, aimed at security and operational testing of optical interfaces, has been causing quite a stir. Lots of vendors and utilities have been in touch to hear more about the product and the capabilities it brings to bear.

We are pleased with the interest in the PP4SM release and happy to discuss some of our further plans for future ProtoPredator products. The idea is for the ProtoPredator line to expand into a family of products aimed at giving developers, device designers and owners/operators a tool set for doing operational and security testing. We hope to extend the product family across a range of ICS protocols. We are currently working on a suite of ProtoPredator tools in our testing lab, even as we “scratch our own itch” and design them to answer the needs we have in performing testing of SmartGrid and other ICS component security assessments and penetration tests.

Thanks to the community for their interest in ProtoPredator. We have a lot more to come and we greatly appreciate your support, engagement and feedback.

ProtoPredator for Smart Meters Released

Today, MicroSolved, Inc. is proud to announce the availability of their newest software product – ProtoPredator™ for Smart Meters (PP4SM). This tool is designed for smart meter manufacturers, owners and operators to be able to easily perform security and operational testing of the optical interfaces on their devices.

PP4SM is a professional grade testing tool for smart meter devices. Its features include:

  • Easy to use Windows GUI
  • Easy to monitor, manage and demonstrate testing to management teams
  • Packet replay capability empowers testers to easily perform testing, verification and demonstrations
  • Manual packet builder 
  • Packet builder includes a standards compliant automated checksum generator for each packet
  • Automated packet session engine 
  • Full interaction logging
  • Graphical interface display with real time testing results, progress meters and visual estimations
  • Flexibility in the testing environment or meter conditions

The tool can be used for fuzzing smart meter interactions, testing protocol rule enforcement, regression testing, fix verification and even as a mechanism to demonstrate identified issues to management and other stakeholders. 

ProtoPredator for Smart Meters is available commercially through a vetted licensing process. Licenses are available to verified utilitiy companies, asset owners, asset operators and manufacturers of metering devices. For more information about obtaining PP4SM or to learn more about the product, please contact an MSI account representative. 

More information is available via:

Twitter: @lbhuston

Phone: (614) 351-1237 ext 206

Email: info /at/ microsolved /dot/ com (please forgive the spam obfuscation…) 🙂

Thanks for Another Great ICS/SCADA Security Symposium

 

J0289528

Thanks to all who helped make the ICS/SCADA Security Symposium fantastic again this year. Great conversations, excellent content and such friendly discussions among peers and concerned parties. 

Next year, we plan to open the event to attendees from throughout the midwest and hope to get even more participation from manufacturing and those who support critical infrastructures. 

Thanks again for all of the hard work that Connie, Chris and the rest of the organizers did to make the event possible. Most of all, thank you for attending, participating and trusting us (and each other) to create such an amazing process of open dialogue. You are all heroes in my book!

ICS/SCADA Security Symposium Reminder

COLUMBUS, Ohio October 9, 2012 – The second annual ICS/SCADA Security Symposium, to be held November 1 2012 in Columbus, is designed to serve as a level set for teams and organizations who are actively managing production ICS/SCADA environments. Once again, this full day session will include best practices advice, incident response, detection techniques and a current threat briefing focused on ICS/SCADA providers. Presenters will cover a variety of topics about what is working, what is not working so well in terms of information security, network protection and trust management. To learn more about the event and to see if you qualify to attend, please contact us via email (info<at sign>microsolved(<dot>)com) or via phone by calling 614.351.1237 ext 215. Chris Lay (@getinfosechere) is handling the invitee list for the event and will be happy to discuss the event with you in more detail. Attendance is free of charge, meals will be provided and a limited number of seats are still available if you qualify.

MSI Announces The Second Annual ICS/SCADA Security Symposium

COLUMBUS, Ohio October 9, 2012 – The second annual ICS/SCADA Security Symposium, to be held November 1 2012 in Columbus, is designed to serve as a level set for teams and organizations who are actively managing production ICS/SCADA environments. Once again, this full day session will include best practices advice, incident response, detection techniques and a current threat briefing focused on ICS/SCADA providers. Presenters will cover a variety of topics about what is working, what is not working so well in terms of information security, network protection and trust management. To learn more about the event and to see if you qualify to attend, please contact us via email (info<at sign>microsolved(<dot>)com) or via phone by calling 614.351.1237 ext 215. Chris Lay (@getinfosechere) is handling the invitee list for the event and will be happy to discuss the event with you in more detail. Attendance is free of charge, meals will be provided and a limited number of seats are still available if you qualify.