See You At EPRI Event in Chicago

Next Monday, June 17th, I’ll be presenting at the EPRI conference in Chicago. My topic is a threat update on what attackers are targeting and what kind of value future state designs and other research/planning data has on the attacker market. If you’re going to be at the event, please join me for my presentation. If you’d like to grab a coffee or the like, let me know. I’ll be around all day. 

Thanks for reading and I hope to see you there! 

Fuzzing Optical Smart Meters with ProtoPredator

PPClawsWords1

Our team has been working hard in the lab, once again testing the optical implementations of a variety of smart meters. Using our proprietary in-house developed tool, called ProtoPredator for Smart Meters, we have been doing full fuzzing of optical protocol implementations. 

Our tool makes this process easy and reproducible. It also provides for easy regression testing and fix validation through session replays. 

One of the things that makes ProtoPredator so cool is that it includes both arbitrary conversations with the meters in addition to canned sessions, making much more flexible in the hands of a knowledgeable user. You can easily use this feature to perform more nuanced validation of the protocols, testing things like sequence errors, poor trust, error recovery, etc. 

While ProtoPredator is still tied to the optical coupler speed and the inherent speed of the protocols in use, testing with it makes validation of the optical ports more effective than other more traditional approaches. Additionally, you can use multiple seats of ProtoPredator in parallel to decrease the overall testing and validation time, especially since the “brain files” and packet sessions are easily interchangeable amongst installations.

The easy to use GUI also means less frustration and more time on task for most users. It lets the testers spend less time on mundane tasks like serial configuration and hand crafting packets and more time on security testing, protocol analysis and bug hunting.

To find out more about ProtoPredator, or to discuss having our lab give your smart meters a look over, get in touch. Info(at)micro solved(dot)com will get you a prompt response. As always, thanks for reading! 

IT/OT/Business Integration Insights from ComEd

Background:

For several years now I have been working with utility companies, and other critical infrastructure organizations particularly focused on Industrial Control Systems (ICS) and Operations Technology (OT) solutions such as SCADA. During that time, one of the most common issues that our customers and the folks who attend our Security Summit every Fall discuss with us revolves around a lack of communication, engagement and ultimately cooperation between ICS engineers, along with Operations staff and the more traditional enterprise focused IT teams. In many cases, this is often expressed as the number one issue that the organization faces.

 

A few years ago, I began asking around the community who might have a solution to this problem. Several people pointed me in the direction of Commonwealth Edison Co. (ComEd), the electric utility in Illinois, which led me eventually to a gentleman named Mark Browning. Through a mutual business partner, I asked to be introduced to Mark, and during that introduction, asked  if he would agree to discuss this problem and the methods ComEd has used to tackle it. Thankfully, Mark and his team agreed. What follows is a summary of the information I gathered from several email interviews and time spent with Mark on the phone.

 

A Bit About Mark:

The first thing you should know is that Mark is a seasoned veteran of the ICS and OT world. He has spent an entire career working in IT, Operations Support and other functions in the ComEd utility. He is, by his own admission, an “old school SCADA” guy. Over the years he has moved from designing and implementing ICS and OT systems through the ranks of  OT application support and eventually into a leadership position where he oversees both traditional IT and the OT teams. It is this experience, along with the commitment, passion and wisdom of the entire ComEd team that make them successful at tackling what seems to be such an industry wide problem.

 

A Bit About ComEd and Exelon:

ComEd is an energy delivery company providing electric transmission and distribution services in the northern 3rd of Illinois, including the Chicago metropolitan area. Exelon Corporation is the parent company of ComEd. As part of Information Technology, Mark and his team work for a corporate shared services group, Exelon Business Services Company.  Mark’s Utility Solutions team  is responsible for the successful implementation and management of IT and OT architectures across and throughout the utility lines of business of ComEd. Embedded in the ComEd business to be close to their counterparts, Mark and his team are directly focused on the success of the business and on providing support to each of those business lines of his customers. This client focused business model is one of the things that Mark credits with keeping his team actively engaged with his business partners and not just supporting requests – thus truly empowering each of the lines of business.

 

This organizational design creates a system of centralized leadership for IT and OT technologies. Acting as a centralized technology group, Utility Solutions is responsible for service levels across all business functions. By design, this creates a direct chain of responsibility to each of the lines of business, and makes technology success fully dependent on the success of each line of business. Mark says this level of integration fully supports solving the lack of engagement problem.

 

How Does It Work at ComEd?:

Mark and his team shared that the strength of engagement between the IT and Business teams stems from a program created more than 10 years ago. They call it the “client engagement model”. Basically, it is a process of fully embedding IT alongside the lines of business. While IT and the Business perform their respective roles, they also collaborate heavily to achieve common objectives. This has created an atmosphere of respect and trust between groups who are comfortable with the shared vision of business goals and an open architecture roadmap to support those goals both short and long-term.

 

In order to cement and maintain that trust between the lines of business and the technology teams, all projects require co-sponsorship and co-leadership. Representatives work directly with their embedded team members in order to create, lead, implement and manage the projects required to build each line of business. Mark’s team members emphatically shared, via a variety of emails, how much easier it makes the job of doing IT well using this approach. They raved about their relationships with the lines of business, with their business focused teammates and with the upper management and leadership of their organization. In particular, many of them commented on how refreshing it was to get to see the technology products that they created actually in use in the business and serving the needs of the end users.

 

It should be noted that such trust between technology teams and lines of business would be nearly impossible to build were it not for a laser-like focus on business problems. Team members with strong technical skills must interface directly with business team members who have strong organizational and communication skills. The problems of the business must be clearly and concisely expressed between the teams and there must be full integration between technology teams and the lines of business. Mark credits much of the success of this program with the embedded nature, that is putting IT and OT people directly in everyday contact with their business partners focused on each line of business.

 

What Can You Do?:

I asked Mark what lessons could be learned from the ComEd approach. In order to help other folks who might not have 10 years of  inertia behind them, I asked Mark what are the key things he would do to apply a similar program to a new organization just beginning to tackle this problem. Mark shared with me the following four key undertakings:

  • Immediately and fully embed and co-locate the IT staff with the business staff members . Ensure that all projects begin to be co-led by a member of the IT team and the business team. Make both of the teams directly responsible for the success of projects.
  • Increase cross training and shared knowledge between the two groups who are now embedded together. Make sure that you are hiring great leaders, and where possible, hire from within the lines of business. Consider functional swaps, where traditional IT staff members temporarily swap positions with business team members. This system of functional swaps often leads to rapid cross communication and knowledge sharing between teams on both a functional and personal level.
  • Hammer home the idea of customer facing trust and co-working communications. Active engagement must occur at all levels for maximum success.  From VP to individual contributor, the IT and business teams must challenge their counterparts by being both advocates and challengers.  Include a shared mission message along the lines of “we must work together because our customers expect us to do so”. Make this mantra a part of everyday life for all team members.
  • Greatly increase the amount of coaching and management level engagement across the now embedded teams. Especially engage in ongoing training for technical team members to see, feel and engage in business operations. Encourage opportunities for the business to directly demonstrate how technology products support both the business and the customer. Clearly demonstrate the benefits to both teams of working together to provide value to the customer.

 

The Payoff:

Lastly, I asked Mark about the payoff for organizations who successfully increase the cooperation and engagement of their IT and business teams. Mark and I both agreed that as the convergence between information technologies and utility delivery mechanisms increase, so too does the importance of integrating these teams.  Essentially, Mark believes that IT has quite a bit to bring to the table.  “IT will become the engine of the utility.”, says Mark. While we both  agree that security remains a risk that we are carrying, convergence and automation will create a unique opportunity to work together to protect and support both the goals of the business,  the desires of the customer and the public at large. With technologies like smart grid on the horizon, those organizations that can effectively conquer the problem of IT and business engagement will be the leaders for the utility markets of the future.

 

Thanks:

I would like to thank Mark and the teams at both ComEd and Exelon for their willingness to discuss their program and to help others with one of the biggest problems many organizations face today. I hope you enjoyed learning from their experiences, and both Mark and I hope that it helps your organization. As always, thanks for reading and until next time, stay safe out there!

Remember Public Cellular Networks in Smart Meter Adoption

One of the biggest discussion points at the recent MEA Summit was the reliance of Smart Meter technology on the public cellular networks for communication.

There seemed to be a great deal of confusion about negotiating private cellular communications versus dependence on fully public networks. Many folks also described putting in their own femtocell and microcell deployments to greatly reduce the dependence on communication assets that they did not own. However, as you might expect, the purchase, install, management, and maintenance of private cellular infrastructure is expensive, requires skilled personnel, and often bumps into regulatory issues with frequency control and saturation.

Other considerations than cost also emerged with several ICS/SCADA owners discussing prioritization of repair issues versus consumer deployments, problems with negotiating effective, acceptable Service Level Agreements with the cell network vendors and a lack of understanding on the cell vendors’ part about ICS/SCADA deployments/integration/criticality in general.
 
Clearly, more analysis, study, and communication needs to occur between ICS/SCADA researchers/owners/developers and the relevant cellular network engineers/implementation teams to grow mutual knowledge and understanding between the parties. In the meantime, ICS/SCADA owners must strive to clearly identify their needs around cellular technologies, clearly demarcate the requirements for private/segmented/public cellular network use and understand the benefits/issues and threats of what they are utilizing. Cellular communications has a clear role to play in the future of ICS/SCADA, but the waters of how it will be managed, how it will be secured and how smaller organizations can obtain it affordably remain a bit muddy for now.
 
If your organization has winning strategies or has concerns that have arisen with the use of cellular networks, we would love to hear about them in the comments. The more ICS/SCADA owners work together to bring this knowledge forward, the more quickly and effectively we can resolve many of the issues that utilities and other organizations are encountering.

Getting Your ICS/SCADA Components Security Tested

Recently, at the MEA Summit, I had the opportunity to engage in a great discussion with a number of SCADA owners about security testing of their devices. Given all of the big changes underway concerning SCADA equipment, connectivity and the greater focus on these systems by attackers; the crowd had a number of questions about how they could get their new components tested in a lab environment prior to production deployment.

Device and application testing is something that MicroSolved has done for more than a decade. We have tested hundreds of IT hardware products, commercial software loads, web/mobile applications, consumer products, and for the last several years, ICS/SCADA and Smart Grid components. Our lab environments are suitable for a wide variety of testing scenarios and are used by utility companies, manufacturers and software developers from around the world as a trusted source for rational security testing and relevant threat analysis. We have a firm non-disclosure policy for client systems tested and the relevant vulnerabilities discovered and we often work hand in hand with the developers/design engineers to work through both mitigation and/or compensating control development.
 
ICS/SCADA owners should have any new designs assessed prior to implementation, they should have some form of ongoing security assessment (analysis – NOT scanning…) performed against current deployments/threats, plus they should be engaged in testing all new hardware and software platforms before production adoption. Developers, designers and manufacturers of ICS/SCADA/Smart Grid components should be engaging in a full set of product assessments, attack surface analysis, threat modeling and penetration testing prior to the release of the products to market. This will be a value-add to your customers, and ultimately, to the consumer. 
 
If your organization would like to have a device or software analysis performed, or would like to discuss how to engage with MicroSolved to have new equipment or ICS/SCADA deployment ideas modeled, tested and assessed, please contact us. 

MSI Strategy & Tactics Talk Ep. 13: SCADA & Handling Threats In a Post-Stuxnet World


SCADA is becoming a hot property among security professionals who work with Industry Control Systems (ICS). During this discussion, our team tackles how to view threats and respond accordingly. Discussion questions include:

  • How can organizations get their heads wrapped around what it takes to secure a modern SCADA/Business environment hybrid?
  • What happened to the air gap approach that we hear so many SCADA history folks talk about? Why did that model break down? Why can’t we go back to it?
  • What happens to threats against SCADA/ICS as mobile integration, smart grid components and other disruptive technologies come online?
  • How can SCADA/ICS security teams engage with other security professionals and each other?

 
Panelists:
Brent Huston, CEO, Founder, and Security Evangelist
Adam Hostetler, Network Engineer and Security Analyst
Phil Grimes, Security Analyst
John Davis, Risk Management Engineer
Mary Rose Maguire, Marketing Communication Specialist and moderator

Click the embedded player to listen. Or click this link to access downloads. Stay safe!

MSI Announces The Ohio SCADA Security Symposium

The need for the latest information about SCADA/ICS is extended to Ohio businesses and utility companies and supports security for Ohio. We’d like to invite all Ohio SCADA/ICS professionals to attend this free event!

The Ohio SCADA Security Symposium, to be held on November 1, 2011 in Columbus, Ohio, is designed to serve as a level set for teams and organizations who are actively managing production SCADA and Industrial Control System (ICS) environments in Ohio.

A full one day session will include best practices advice, incident response, detection techniques and a current threat briefing focused on SCADA/ICS providers. Presenters will cover a variety of topics about what is working and what is not, in terms of information security, network protection and trust management.

Takeaways from this event will include peer networking, insights into emerging threats, action items for actively improving the availability, integrity and confidentiality of control systems, utility networks, manufacturing lines and other SCADA/ICS concerns.

Topics include: How the State Is Here to Help You, Physical Security, Assessment of SCADA/ICS Environments, Cyber Security, Honey Pots in SCADA/ICS Environments, and The FBI Viewpoint. Key participation will feature NiSource, American Electric Power, American Municipal Power, Greater Cincinnati Water Works, Ohio PUCO, the Department of Homeland Security, and the FBI.

The event runs from 8:30 AM to 6:00 PM. Registration opens at 8:00 AM and is free. Those who work with SCADA/ICS are invited to attend. RSVP’s can be sent to mmaguire@microsolved.com. Please include your contact information. Seating is limited and available ONLY to those individuals actively working in Ohio with SCADA/ICS components.

MSI looks forward to providing an excellent event that will help organizations secure their SCADA/ICS systems and discuss best practices and industry standards at the event!