Thanks for Making the 3rd Mid-West ICS/SCADA Security Symposium a Success

Thanks to the attendees and speakers who participated yesterday in the 3rd Annual ICS/SCADA Security Symposium. It was another great event and once again, the center of the value was in the interactions of the audience with the speakers and each other. It’s great to hear asset owners discuss what is working, what is challenging and what is critical in their minds.

Thanks again to those who attended and contributed to making this event such a wonderful thing again this year. We appreciate it and we can’t wait until next year to do it all again.

Thank YOU!

Save The Date: Midwest ICS/SCADA Security Symposium 2013

Just a quick announcement that the 3rd annual Midwest ICS/SCADA Security Symposium date has been announced. We will be holding the event on November 14th, 2013 in Columbus, Ohio.

It is a single track, single day event which is highly focused on peer to peer interaction between asset owners, utilities, manufacturers and other interested parties. The attendees usually span the various types of ICS asset holders from water, power, natural gas, chemical, automated manufacturing and other critical infrastructures. The focus is on real world threats, changing regulatory guidance, what controls work and work less, scenarios and tactics that have helped improve security and overall changes in protection strategies in the last 12 months.

The conversations are often candid, to the point and the open forum leads to passionate and real world discussions.

All attendees are vetted to ensure confidentiality and maintain focus on real content minus vendor sales pitches. The cost to attend is FREE and coffee, snacks and lunch is provided.

To learn more about the event or to qualify for an invitation, please drop us a line via email (info A T microsolved D O T com) or via aTwitter (@lbhuston or @microsolved). If you have attended or qualified in the past for the event, your invitation will be forthcoming shortly.

Speaker selection is now underway, so watch this blog for the agenda in the near future. 

Cyber News Today from Homeland Security Middle East – Abu Dhabi, UAE

Happy Memorial Day Readers;

The Red Dragon and MicroSolved are at the Homeland Security Summit- Middle East taking place in Abu Dhabi, United Arab Emirates…

Latest World Cyber News you should be maintaining cyber situational awareness on comes to you today after 6 different flights across 4 different continents and a total of 30,000 airmiles…oh yes 5 hours of sleep –

Nonetheless – here are some developing stories out of the International Cyber World….

General Alexander – Four-star general in eye of U.S. cyber storm… Read more @ http://newsle.com/article/0/76523525/

The covert battle over Beijing’s defence policy heats up…People’s Republic of China gets into the business of making friends

Read more @ http://www.smh.com.au/world/china-gets-into-the-business-of-making-friends-20130524-2k6q3.html#ixzz2UTeO2Fht

People’s Republic of China’s Huawei a victim of its success

Read more @: http://www.chinadaily.com.cn/cndy/2013-05/25/content_16530834.htm &
http://wanderingchina.org/2013/05/26/huawei-a-victim-of-its-success-china-daily-risingchina-trade/

All for now from the Middle East…more to come as the world wakes to a new day…

Semper Fi,

謝謝

紅龍

What YOU Can Do About International Threats

Binary eye

With the addition of RedDragon Rising (@RedDragon1949) to the blog, we are now pushing forth a new stream of threat data and insights about the growing problem of international threats. Since we added that content to the site, many of you have written in or asked me on Twitter, what is it that YOU can do about these threats? I wanted to take a few minutes and expand on my responses.

First of all, you can remain aware and vigilant. Much of the information we post here isn’t directly actionable. It isn’t designed to be a roadmap of actions for you to take. It’s designed to be a continual source of data that slowly helps you see a clearer picture of the threat, the actors and their capability. It’s designed to keep you AWAKE. It’s custom made to help you understand your adversary. Knowledge is power and insight is key. We make this content to give you both!

Second, you can communicate the threat and knowledge to your management. This helps them remain aware. It also presents to them that you are monitoring the threats and keeping your eye on the rising tides, even as you help them steer the ship through safe waters. You can use this information to build rapport with them, to give them new insights into your decisions when you explain to them various risks and to help them understand the changing nature of the interconnected world.

You can use the information here as an impetus to get the basics of information security right. While there aren’t any panaceas to fight off the threat and there isn’t a single thing you can buy to make it better ~ we do know that focusing on the basics of infosec and getting them done efficiently, effectively and well is the best defense against a variety of threats. That said, consider doing a quick and dirty review of your security initiatives against our 80/20 Rule for Information Security. This is a set of simple projects that represent the basics of information security and map easily to other standards and baselines. Simply judging your maturity in these areas and following the roadmap to improvement will go a long way to getting the basics done right in your organization. 

Invest in detection and response. If your organization is doing the basics of prevention, that is you have hardening in place and are performing ongoing assessment and mitigation of your attack surfaces, then the next thing to do is invest in detection and response capabilities. Today, one of the largest advantages that attackers enjoy is the lack of visibility and effective response capabilities in our organizations. You should have some visibility into every segment and at every layer of your environment. You should be able to identify compromises in a timely manner and move to isolate, investigate and recover from any breaches LONG BEFORE they have become widespread and heavily leveraged against you. If you can’t do that today, make it your next major infosec goal. Need help?Ask us about it.

Lastly, share information with your peers. The bad guys are good at information sharing. They have excellent metrics. They openly share their experiences, successes, failures and new techniques. Much of crime and espionage (not all, but MUCH) is “open source” in nature. The cells of attackers free float in conglomerations of opportunity.  They barter with experience, tools, data and money. They share. The more we begin to share and emulate their “open source” approaches, the better off we can be at defending. If knowledge is power, more brains with more knowledge and experience equals MORE POWER. Be a part of the solution.

That’s it for now. Just remain calm, get better at the basics, improve your visibility and stay vigilant. As always, thanks  for reading State of Security and for choosing MicroSolved as your information security partner. We are striving to dig deeper, to think differently and to give you truly actionable intelligence and threat data that is personalized, relevant to your organization and meaningful. If you’d like to hear more about our approach and what it can mean for your organization, get in touch via Twitter (@lbhuston), email (info(at)microsolved/dot/com) or phone (614-351-1237 ext 250). 

Fuzzing Optical Smart Meters with ProtoPredator

PPClawsWords1

Our team has been working hard in the lab, once again testing the optical implementations of a variety of smart meters. Using our proprietary in-house developed tool, called ProtoPredator for Smart Meters, we have been doing full fuzzing of optical protocol implementations. 

Our tool makes this process easy and reproducible. It also provides for easy regression testing and fix validation through session replays. 

One of the things that makes ProtoPredator so cool is that it includes both arbitrary conversations with the meters in addition to canned sessions, making much more flexible in the hands of a knowledgeable user. You can easily use this feature to perform more nuanced validation of the protocols, testing things like sequence errors, poor trust, error recovery, etc. 

While ProtoPredator is still tied to the optical coupler speed and the inherent speed of the protocols in use, testing with it makes validation of the optical ports more effective than other more traditional approaches. Additionally, you can use multiple seats of ProtoPredator in parallel to decrease the overall testing and validation time, especially since the “brain files” and packet sessions are easily interchangeable amongst installations.

The easy to use GUI also means less frustration and more time on task for most users. It lets the testers spend less time on mundane tasks like serial configuration and hand crafting packets and more time on security testing, protocol analysis and bug hunting.

To find out more about ProtoPredator, or to discuss having our lab give your smart meters a look over, get in touch. Info(at)micro solved(dot)com will get you a prompt response. As always, thanks for reading! 

IT/OT/Business Integration Insights from ComEd

Background:

For several years now I have been working with utility companies, and other critical infrastructure organizations particularly focused on Industrial Control Systems (ICS) and Operations Technology (OT) solutions such as SCADA. During that time, one of the most common issues that our customers and the folks who attend our Security Summit every Fall discuss with us revolves around a lack of communication, engagement and ultimately cooperation between ICS engineers, along with Operations staff and the more traditional enterprise focused IT teams. In many cases, this is often expressed as the number one issue that the organization faces.

 

A few years ago, I began asking around the community who might have a solution to this problem. Several people pointed me in the direction of Commonwealth Edison Co. (ComEd), the electric utility in Illinois, which led me eventually to a gentleman named Mark Browning. Through a mutual business partner, I asked to be introduced to Mark, and during that introduction, asked  if he would agree to discuss this problem and the methods ComEd has used to tackle it. Thankfully, Mark and his team agreed. What follows is a summary of the information I gathered from several email interviews and time spent with Mark on the phone.

 

A Bit About Mark:

The first thing you should know is that Mark is a seasoned veteran of the ICS and OT world. He has spent an entire career working in IT, Operations Support and other functions in the ComEd utility. He is, by his own admission, an “old school SCADA” guy. Over the years he has moved from designing and implementing ICS and OT systems through the ranks of  OT application support and eventually into a leadership position where he oversees both traditional IT and the OT teams. It is this experience, along with the commitment, passion and wisdom of the entire ComEd team that make them successful at tackling what seems to be such an industry wide problem.

 

A Bit About ComEd and Exelon:

ComEd is an energy delivery company providing electric transmission and distribution services in the northern 3rd of Illinois, including the Chicago metropolitan area. Exelon Corporation is the parent company of ComEd. As part of Information Technology, Mark and his team work for a corporate shared services group, Exelon Business Services Company.  Mark’s Utility Solutions team  is responsible for the successful implementation and management of IT and OT architectures across and throughout the utility lines of business of ComEd. Embedded in the ComEd business to be close to their counterparts, Mark and his team are directly focused on the success of the business and on providing support to each of those business lines of his customers. This client focused business model is one of the things that Mark credits with keeping his team actively engaged with his business partners and not just supporting requests – thus truly empowering each of the lines of business.

 

This organizational design creates a system of centralized leadership for IT and OT technologies. Acting as a centralized technology group, Utility Solutions is responsible for service levels across all business functions. By design, this creates a direct chain of responsibility to each of the lines of business, and makes technology success fully dependent on the success of each line of business. Mark says this level of integration fully supports solving the lack of engagement problem.

 

How Does It Work at ComEd?:

Mark and his team shared that the strength of engagement between the IT and Business teams stems from a program created more than 10 years ago. They call it the “client engagement model”. Basically, it is a process of fully embedding IT alongside the lines of business. While IT and the Business perform their respective roles, they also collaborate heavily to achieve common objectives. This has created an atmosphere of respect and trust between groups who are comfortable with the shared vision of business goals and an open architecture roadmap to support those goals both short and long-term.

 

In order to cement and maintain that trust between the lines of business and the technology teams, all projects require co-sponsorship and co-leadership. Representatives work directly with their embedded team members in order to create, lead, implement and manage the projects required to build each line of business. Mark’s team members emphatically shared, via a variety of emails, how much easier it makes the job of doing IT well using this approach. They raved about their relationships with the lines of business, with their business focused teammates and with the upper management and leadership of their organization. In particular, many of them commented on how refreshing it was to get to see the technology products that they created actually in use in the business and serving the needs of the end users.

 

It should be noted that such trust between technology teams and lines of business would be nearly impossible to build were it not for a laser-like focus on business problems. Team members with strong technical skills must interface directly with business team members who have strong organizational and communication skills. The problems of the business must be clearly and concisely expressed between the teams and there must be full integration between technology teams and the lines of business. Mark credits much of the success of this program with the embedded nature, that is putting IT and OT people directly in everyday contact with their business partners focused on each line of business.

 

What Can You Do?:

I asked Mark what lessons could be learned from the ComEd approach. In order to help other folks who might not have 10 years of  inertia behind them, I asked Mark what are the key things he would do to apply a similar program to a new organization just beginning to tackle this problem. Mark shared with me the following four key undertakings:

  • Immediately and fully embed and co-locate the IT staff with the business staff members . Ensure that all projects begin to be co-led by a member of the IT team and the business team. Make both of the teams directly responsible for the success of projects.
  • Increase cross training and shared knowledge between the two groups who are now embedded together. Make sure that you are hiring great leaders, and where possible, hire from within the lines of business. Consider functional swaps, where traditional IT staff members temporarily swap positions with business team members. This system of functional swaps often leads to rapid cross communication and knowledge sharing between teams on both a functional and personal level.
  • Hammer home the idea of customer facing trust and co-working communications. Active engagement must occur at all levels for maximum success.  From VP to individual contributor, the IT and business teams must challenge their counterparts by being both advocates and challengers.  Include a shared mission message along the lines of “we must work together because our customers expect us to do so”. Make this mantra a part of everyday life for all team members.
  • Greatly increase the amount of coaching and management level engagement across the now embedded teams. Especially engage in ongoing training for technical team members to see, feel and engage in business operations. Encourage opportunities for the business to directly demonstrate how technology products support both the business and the customer. Clearly demonstrate the benefits to both teams of working together to provide value to the customer.

 

The Payoff:

Lastly, I asked Mark about the payoff for organizations who successfully increase the cooperation and engagement of their IT and business teams. Mark and I both agreed that as the convergence between information technologies and utility delivery mechanisms increase, so too does the importance of integrating these teams.  Essentially, Mark believes that IT has quite a bit to bring to the table.  “IT will become the engine of the utility.”, says Mark. While we both  agree that security remains a risk that we are carrying, convergence and automation will create a unique opportunity to work together to protect and support both the goals of the business,  the desires of the customer and the public at large. With technologies like smart grid on the horizon, those organizations that can effectively conquer the problem of IT and business engagement will be the leaders for the utility markets of the future.

 

Thanks:

I would like to thank Mark and the teams at both ComEd and Exelon for their willingness to discuss their program and to help others with one of the biggest problems many organizations face today. I hope you enjoyed learning from their experiences, and both Mark and I hope that it helps your organization. As always, thanks for reading and until next time, stay safe out there!

3 Tough Questions with Chris Jager

Recently, I got to spend some time interviewing Chris Jager via email on industrial control systems security. He didn’t pull any punches and neither did I. Here, are 3 Tough Questions between myself (@lbhuston) and Chris.


A Short Biography of Chris Jager (@chrisjager): I have over 15 years of experience in Information Technology and have focused on the practical application of security principles throughout my career. Most recently, I was director of the NESCO Tactical Analysis Center at EnergySec; a non-profit organization formed to facilitate information sharing, situational awareness, and education outreach to the energy sector. I am active in a number of information security workgroups and have provided operational, architectural, and regulatory compliance guidance to large and small organizations in both the public and private sectors, focusing on the energy sector exclusively since 2006.


Brent: You have spent a lot of time working on Industrial Control Systems (ICS) in your career. During that time, you have been witness to the explosion of interest in IT security as a profession. Why should some of the younger folks thinking about information security as a career consider a focus on ICS and SCADA? Why should they care?

Mr. Jager: This is a fantastic question and, if I frame my response correctly, the answer will hopefully be self-evident to your readers.

ICS and SCADA are terms that are seldom understood and often misused by information security (infosec) publications. SCADA systems typically manage geographically disperse areas and often consist of numerous functionally disparate processes.

However, because of the immense variety of different processes that can be managed by industrial control systems, ICS has become somewhat of a catchall term – including SCADA systems. For example, you’ll often find electric power generation processes such as turbine control, burner management, vibration monitoring and more lumped into the mix. Each of these processes has discrete or locally distributed control and instrumentation systems, any of which can cause catastrophic safety, reliability, and financial issues if misused.

For me, the challenge of protecting these kinds of systems is far more interesting than making sure that little Bobby can’t drop the student records table in a classroom database. Much of the actual management technology is the same as what is used in general IT, but the application is very different. Things get a little more exotic (and arcane) when you go further down the stack into digital–to-analog conversion, but it’s not overly difficult for most folks to understand once exposed to it. The negative impacts of misuse aren’t limited to convenience and financial loss. Risk to life and limb is a very real possibility in many processes that are managed by industrial control system automation that is being run out of specification.

Typically, industrial control systems are deployed in step with the physical equipment they are designed to manage. The physical equipment is often orders of magnitude more expensive than the ICS components that ship with it and may be designed for lifespans measured in decades. In short, upgrades seldom occur as they need to be engineered and tested for functionality, safety, and a myriad of other issues pertaining to the existing physical equipment.

This has led to a situation where the groups that understand control systems and processes are naturally (and often generationally) gapped from those groups who understand the current threat and vulnerability landscapes. Consequently, there are currently very few individuals that understand industrial control system security as it relates to the changing threat picture. If the challenge of doing something very few dare to try doesn’t sound good on its own, this is the sound of opportunity knocking. Answer the door!

I’d like to make one last point on this question. Take a look around your house or apartment and count the number of internet-enabled devices you have. Most people these days have far fewer traditional computers than embedded systems – devices that aren’t user-serviceable without breaking a warranty or two. And the hacking skills necessary to modify such devices to fit use cases unintended by the manufacturers seem to come naturally to the younger folk of today. Those skills are also relatively portable to the ICS/SCADA world where embedded systems are the norm. Sure, some of the protocols and hardware packages are somewhat different, but they are all relatively simple compared to what folks are tinkering with at their coffee tables. We can always use more QA/breakers – particularly at key points in the supply chain where issues can be spotted and fixed before they become permanently vulnerable installations. Again I say, “knock knock”!

 

Brent: You talk a lot about how challenging ICS/SCADA security is. Do you think protecting ICS/SCADA systems in a meaningful way is an attainable goal? Assuming yes, do you think it could be done during what’s left of our careers? Why or Why not?

Mr. Jager: If I didn’t think it was an attainable goal, I’d not be doing the kind of work I’ve done over the past number of years. There are much easier ways to make a buck than to have people who are entrenched in the old way of doing things actively work to prevent you from even introducing discussions about change – let alone actually implementing it!

There is momentum in this area, but much work still needs to be done. Devices still ship from manufacturers with easily discerned hardcoded administration credentials, firmware updates are accepted without challenge and more. Once deployed in the field, user passwords seldom change, vulnerabilities discovered post-installation go unmitigated, and so on.

Because we have all this noise around basic security failures and their associated issues, we don’t yet know what constitutes “meaningful” or “attainable” when we speak of complex industrial control systems. A prime example here is that the electric sector is still using the exact same set of controls and asset scoping for its regulated security standards as when I first started working in the sector in 2006. NERC CIP version 1 was in final draft form, and the current requirements catalog will remain largely unchanged until at least 2015 when and if version 5 becomes effective. There have been minor changes in the interim, but not one that comes remotely close to addressing change in the threat landscape.

Will we ever have a perfect system? No. We do, however, urgently need to stop being complacent about the subject and implement those security measures that we can.

 

Brent: If you had your own ICS system, let’s say you ran Chris’s power company, what would that look like? How would it be protected?

Mr. Jager: It would look very, very “dumb”. Until such time as ICS and other automation technologies are vetted by process engineers – and I’m talking about the entire ICS/automation stack, I would automate only where it was impossible to operate the business or process without it.

It seems to me that we have a major employment problem in this country and no clear path to resolution. Putting some of these people to work securing our industrial control systems is an area where the private sector can help get the country back to work without relying on government funded stimulus packages. An added bonus is that we’ll end up with a whole cadre of workers who have been exposed to the industry, a percentage of who will stay in the field and help to address the industry’s gray out problem. All it takes is one or two sizable impacts from automation failure or misuse for the cost savings seen through automation to be wiped out.

Where I had no choice but to automate, Chris’ Power Company would look very much like any power company out there today, unfortunately. There simply aren’t enough vendors and manufacturers out there presently that produce secure equipment. Even then, systems integrators often further weaken the environment by adding support accounts and other remotely accessible backdoors to these systems.

Be it in the energy sector or any other, process automation installations will inevitably mature to a state of persistent vulnerability due to their long lifespans. Vulnerability discovery and exploitation techniques advance over time, vulnerabilities are introduced through regression bugs elsewhere in the software or protocol stack, or the process itself may have changed to a point where a previously innocuous vulnerability now has the ability to introduce a large impact if exploited.

Eventually, pointing out that the emperor has no clothes becomes a career limiting move – particularly when said emperor is an exhibitionist! Instead, the focus should be on identifying the more sensitive individuals in the crowd and protecting them appropriately through sound risk identification principles. We can’t make the problems go away through risk management, but we can use the techniques to identify the things that matter most and, where we can’t mitigate the risk, implement monitoring and response controls. This sort of approach also helps prioritize future efforts and dollars.

The top security controls at Chris’ Power Company would center around monitoring and response as employees would be trained to assume the environment was in a persistent state of compromise. In the environment we live in today where threats are real and expressed, and vulnerabilities aren’t able to be universally mitigated, the only real chance at controlling risk you have is to manage the impact of a successful attack. You only get that chance if you are able to detect and respond before the attack balloons to the maximum impact value.

If you failed to give my company that chance, you wouldn’t be working at Chris’ Power Company!


Thanks to Chris Jager for his insights and passion about ICS security. We appreciate his willingness to spend time with us. Thanks, as always, to you the reader, for your attention. Until next time, stay safe out there!

HoneyPoint Security Server ICS/SCADA Deployment Example

Recently, there have been several questions about potential deployment scenarios for HoneyPoint Security Server in and around ICS and SCADA organizations. Here is a quick, high level view of what a sample deployment might look like in a utility or other ICS environment. Note that the sample environment has fully embraced enclaveing. The network is fully segmented based on function.

In organizations where segmentation or the use of enclaves has not been established, HPSS can still be used and would be deployed in much the same manner.

Please let us know if you have any questions about this diagram or about deploying HPSS in your environment. We would be happy to set up a free consultation with you to discuss how the tool could aid in your detection program and give you increased visibility throughout your enterprise.

PS – If the graphic is difficult to read, right click on it and select view in new tab. The theme for the site is having trouble with this particular graphic.

HighLevelEnclaves

Threat Data Sharing in ICS/SCADA Needs Improvement

I had an interesting discussion on Twitter with a good friend earlier this week. The discussion was centered around information sharing in ICS/SCADA environments – particularly around the sharing of threat/attack pattern/vulnerability data. 

It seems to us that this sharing of information – some might call it “intelligence”, needs to improve. My friend argues that regulation from the feds and local governments have effectively made utilities and asset owners so focused on compliance, that they can’t spare the resources to share security information. Further, my friend claims that sharing information is seen as dangerous to the utility, as if the regulators ever found out that information was shared that wasn’t properly reported “up the chain”, that it could be used against the utility to indicate “negligence” or the like. I can see some of this, and I remember back to my DOE days when I heard some folks talk along the same lines back when we showed up to audit their environments, help them with incidents or otherwise contribute to their information security improvement.

When I asked on open Twitter with the #ICS/#SCADA hashtags about what hampered utilities from sharing information, the kind Twitter folks who replied talked about primarily three big issues: the lack of a common language for expressing security information (we have some common languages for this (mitre’s work, VERIS, etc.)), legal/regulatory concerns (as above) and the perceived lack of mitigations available (I wonder if this is apathy, despair or a combination of both?). 

I would like to get some wider feedback on these issues. If you don’t mind, please let me know either in comments, via private email or via Twitter (@lbhuston) what you believe the roadblocks are to information sharing in the ICS/SCADA community.

Personally, I see this as an area where a growth of “community” itself can help. Maybe if we can build stronger social ties amongst utilities, encourage friendship and sharing at a social level, empower ourselves with new mechanisms to openly share data (perhaps anonymously) and create an air of trust and equity, we can solve this problem ourselves. I know the government and industry has funded ISACs and other organizations, but it seems to me that we need something else – something more easily participatory, more social. It has to be easier and safer to share information between us than it is today. Maybe, if we made such a thing, we could all share more openly. That’s just my initial 2 cents. Please, share yours.

Thanks for reading, and until next time, stay safe out there!  

ProtoPredator for Smart Meters Released

Today, MicroSolved, Inc. is proud to announce the availability of their newest software product – ProtoPredator™ for Smart Meters (PP4SM). This tool is designed for smart meter manufacturers, owners and operators to be able to easily perform security and operational testing of the optical interfaces on their devices.

PP4SM is a professional grade testing tool for smart meter devices. Its features include:

  • Easy to use Windows GUI
  • Easy to monitor, manage and demonstrate testing to management teams
  • Packet replay capability empowers testers to easily perform testing, verification and demonstrations
  • Manual packet builder 
  • Packet builder includes a standards compliant automated checksum generator for each packet
  • Automated packet session engine 
  • Full interaction logging
  • Graphical interface display with real time testing results, progress meters and visual estimations
  • Flexibility in the testing environment or meter conditions

The tool can be used for fuzzing smart meter interactions, testing protocol rule enforcement, regression testing, fix verification and even as a mechanism to demonstrate identified issues to management and other stakeholders. 

ProtoPredator for Smart Meters is available commercially through a vetted licensing process. Licenses are available to verified utilitiy companies, asset owners, asset operators and manufacturers of metering devices. For more information about obtaining PP4SM or to learn more about the product, please contact an MSI account representative. 

More information is available via:

Twitter: @lbhuston

Phone: (614) 351-1237 ext 206

Email: info /at/ microsolved /dot/ com (please forgive the spam obfuscation…) 🙂