141. Idea - The Partnership Between Safety and Security

Richie: 00:00 

So there are things going on in the real world that affect functional safety and live outside of the understanding of PLCs and safety standards and such, and security is chief among those. If your system is not secure, it is not safe. It doesn't matter how many things you've checked off on your safety checklist, a failure to secure is a safety liability. So we really should be viewing the topics of safety and security as being strongly tied together. 

Chris: 00:25 

Welcome to EECO Asks Why a podcast that dives into industrial manufacturing topics, spotlights the heroes that keep America running. I'm your host, Chris Grainger. And on this podcast, we do not cover the latest features and benefits on products that come to market, instead, we focus on advice and insight from the top minds of industry because people and ideas will be how America remains number one in manufacturing in the world. 

Welcome to EECO Asks Why. Today we have an idea episode and we'll be talking about how you establish a partnership between safety and security. So it's going to be very fun. I'm excited to have with me, Mr. Richie Fortenberry, who is a consulting engineer and operations manager at Prism Systems. So welcome Richie How are you doing?

Richie: 01:15 

I'm well, how are you? 

Chris: 01:16 

I'm good. Now, where are you located at? 

Richie: 01:19 

I'm in Birmingham, Alabama. Our home office is in Mobile, Alabama. We have, we have some people in Orlando and in Glendale area in California and a few in the Detroit are but my office is the Birmingham area. 

Chris: 01:32 

Okay. Great. I love that Alabama. Sure the summertime can get tough, but that will say that for another conversation, man. How about talking about safety and security, we have a lot of listeners out there who may not be familiar with what we're talking about for this particular episodes of get them up to speed. From an industrial standpoint, what are you referring to here? 

Richie: 01:53 

So let's start by contextualizing those terms of safety and security. Those words can be used as a characteristic of a system. Like when we say as a system safer as a system secure, we're talking about characteristics of that system, but in the context I'd like to approach it today, at least initially I think it's helpful to think of those terms as describing processes. 

What steps and methodologies lead to a system being quantifiably safe or secure? So these are the processes of safety and So let's talk about safety first. That's it's probably the more widely understood of the two topics and industry.

So when I talk about safety in the context of control systems, I'm talking about the process of making a system safe, starting with identification and initial assessment of hazards and failure modes assessing the impact of those hazards and failure modes, if they were to be realized. Determining any and applying any mitigations that you might come up with reassessing the system, once everything is mitigated and ultimately implementing those mitigations as design changes or operational or maintenance procedures or actual control system mitigation.

And that's an oversimplification of the safety life cycle, but let's leave it at that. So if this process of safety is successfully executed, then the characteristic of safety can be applied to that system. So think process over characteristic in  this conversation. 

Now onto security security of control systems and OT networks is a similar process to safety in a lot of ways. It's similar actually to securing IT systems and networks or cybersecurity principles more broadly, you know, the same malware that affects IT systems can also bring down an OT system. Many modern control communication protocols are built on top of the same protocols that are widely used in IT applications.

So they, they share the same implicit vulnerabitlities, but that said we should be careful about too closely relating IT and OT security methodologies, OT networks play a bit by their own rules to a degree. IT security places, comparatively higher emphasis on confidentiality and lower emphasis on availability.

If you lose five seconds of availability in an office environment, it's likely that many users might not even notice. It's just a blip in their coverage. In a production environment though, loss of availability for five seconds could incur faults that bring the production line down to cause waste of product, could incur further downtime and attempts to get the system restored back to operation. So OT communications are often deterministic by necessity. You've got IO devices and drives and such that are communicating back and forth with the controller that rely on deterministic communications and generate faults if there are lapses in coverage.

So there's a time sensitivity that drives availability as a key concern and OT environments and that's an important consideration in security. There are also other sensitivities in terms of security, you might go into assess and OT network using your typical tools or tool bag for it security assessment. And you could inadvertently bring an entire production system down by using a port scan or something that's a common and relatively benign tool in other environments, but it could be critical in the OT environment. So OT security is related to IT security and cybersecurity more broadly.

But it's its own thing in a lot of ways. And it merits being considered differently from those other forms of cybersecurity, but on the topic of safety and security taken to. You know that there's a relationship between OT security and cybersecurity as a broad category should be pretty evident immediately, but what's not always evident or are well understood in industry is that there's a strong relationship between security and safety in the OT environments as well.

And that relationship it's kind of been a soap box of mine for awhile, and sometimes it feels like trying to convince people to leave a room that's actively burning down around them. There's all the signs that you would need to point towards a greater emphasis on security and industry. And we've been pretty slow to adopt and to respond to those obvious indicators. 

Chris: 06:12 

I mean, what do you thinks the reason why, the slow response there? I mean, because he definitely laid out some good points here to consider from a safety and security standpoint. Why the slow response?

Richie: 06:22 

I think it's a little bit of complacency and a little bit of a false sense of security based on some insulating factors that we've benefited from and in the OT environment serialized protocols and things like that were, that are not popular elsewhere. I had ran our plants for a long time, but now that we're jumping on the the TCP IP train, but then a lot of our protocols where we've become more exposed.

So we have still a little bit of the idea of, cybersecurity is not really a thing for most production facilities. There's that there's some remnants of that mentality that have been unfortunately hard to route out. 

Chris: 07:01 

Well, I mean, thank you. What you've walked us through there was a lot, definitely to help understand the tie between safety and security. Maybe help us understand a little bit more about the alignment between those two specific to the industrial environment because I know that's where we're trying to focus on with you, Richie. So we're where do you see that, that tight alignment?

Richie: 07:20 

So the alignment between safety and security. So when we're talking about mitigating and responding to hazards and a controlled system in a safety context we're dealing with safety functions on the software side that rely on safety rated hardware and or particular hardware architecture should be effective.

This the safety rating of a piece of hardware or the acceptable architecture that drives a particular safety function. These are defined by a number of industry specifications. ISO 13.8.49 is a popular safety standard. IEC 62.0.61 is another, those are where we get performance level and soul ratings, prospectively there.

And there's also a level of interoperability between those standards and others. So you don't necessarily have to live in one all the time. So we can design these safety systems. We can design them well by applying these specifications and standards and determining our appetite for risk and applying appropriate mitigations.

And we can ultimately quantify how well we're doing in terms of safety using these standards, but we need to understand and that I don't feel is often understood is that these mitigations and these standards and these quantified results that we get back from them, they're all set against the backdrop of things that control systems and relevant safety standards understand.

So we're talking, we're living in at that point in the world and in the context of what PLCs and drives and the standards anticipate and can understand. So we tell these control device is about our world and the things that can go bad in it. And we tell them how to mitigate those things.

And then they just, they go off and do their job, but it's a mistake to place ourselves too firmly, mentally within the constraints of the world, of the control system and these standards, that world is just a loose approximation of our real world. You know the world in which we'd like to keep our bodies, our lives intact.

Yeah so there are things going on in the real world that affect functional safety and live outside of the understanding of PLCs and safety standards and such, and security is chief among those. If your system is not secure, it is not safe. It doesn't matter how many things you've checked off on your safety checklist, a failure to secure is a safety liability. So we really should be viewing the topics of safety and security as being strongly tied together. 

Chris: 09:40 

For sure. And that point of failure to secure is a safety liability, that's it. Love your passion around this topic, and I know we were talking, getting our notes together you mentioned demo and I think this is a great opportunity to bring that up here. And the demo that you built exposes a lot of these areas, those risks and those liabilities, I think you were focusing on the safety PLC. So maybe can you give us walk through what that demo is and how you've used that to educate others around this topic.

Richie: 10:11 

Yeah. It, there's nothing groundbreaking about the demo, but I wanted to demonstrate how safety can be dependent on security and a simple and approachable way. I just thought it'd be helpful to demonstrate for our customers and others. You can talk and talk about a concept, but sometimes it's better to just simply demonstrate that concept in a way that's immediately relatable.

So I built a small panel with a safety PLC inside and some safety IO. Driving four pair of, red and green indicators arranged on the panel door to look like traffic lights at a four way intersection. I didn't use a yellow one because I was cheap. And but but I had it set up where you could look at the front of the panel and immediately  recognize what was going on there. And then I wrote a simple PLC program and the safety task that would turn the lights from red to green, to red, to green in a manner that would simulate a four way traffic light and everything was implemented using a standards-based approach to safety using best practices, using safety rated hardware. So checking all the safety boxes. 

Then I wrote a piece of malware, maybe a hundred lines of C-code using readily accessible libraries that anyone could pull down from the internet to snoop on network traffic and to, to spoof certain network protocols. The malware was capable of scanning a network identifying some of the pieces of control hardware, communicating, using a specific protocol and specifically attacking the relationship between the PLC and the remote IO. And again, this is a safety PLC and these are safety rated remote IO but it would attack it to the effect that it could modify and take control over the producer in that relationship and turn all the lights green at the same time.

So when I would do this in the context of the, of safety, as the control system understands it, all the safety functions were still intact. Nothing was broken there, but in the context of the real world. A very hazardous situation, a very, a catastrophic situation or loss of lives would it be potential in this situation.

We need to be thinking about safety in the real world. And not only in the context of the control system we're bringing ourselves down a level by being exclusive to the things that the standards and the safety processors. Point two, because there they are not in and of themselves security devices.

So the control systems version of safety is an approximation of our own. It's not a direct replica and we need to account for hazards that lie outside of that approximation. And this can't be done without considering security. So again, if a system is not secure, it can't be called safe.

Chris: 12:51 

I'm curious on what, as you've shown that demo to others. What feedback have you gotten? Are you getting a lot of these aha or you gotta be kidding me moments from people? They just, they can't believe it. 

Richie: 13:03 

Yeah, I've had depending on the role of the people that I've showed it to I've had some very technical people say, "Hey, let me, let me see your C-code," and let me, you know, be very interested in and the vulnerabilities of some of those protocols and how difficult or easy it is to crack those.

I presented this at a conference once and had a guy come up and asked me to open the panel because he wanted to, verify that I am actually using the things that I say that I'm using inside the panel. And he was a sharp guy. I talked with him a lot afterwards, but what drove his suspicion is the idea that it can't be that it can't be that, that easy or that, that exposed and yeah, it really is. 

So yeah, the responses has been varied, then all the way to you know, management types that is just an eyebrow raiser and a recognition that, "Hey, we use this stuff, these very same pieces of hardware for critical applications, and we've taken some things for granted that potentially we should not have."

Chris: 14:00 

Yeah. Well, I mean, hats off to you for being innovative and actually bringing that real-world visualization to life to show safety secure. The vulnerabilities that exist. So if somebody is out there listening right now, Richie and you may be, you have their hair standing up on the back of their neck a little bit, they're a little worried about what their risk profile may be. What steps should they take to start understanding that? 

Richie: 14:26 

There's a number of standards out there that provide a roadmap for OT security programs and methodologies. I'm certified in ISA 62.4.43 others that I know that are equally or more proficient even Preparedness 882 as a, starting point.

Those two aren't mutually exclusive. So like the safety standards, there's interoperability between security standards each will point you towards a defense in-depth approach in most cases, which basically means that there are a lot of attack surfaces at different layers in your system. And no one system is a like another our defense strategy should be multi-layered as well. Such that one failure to mitigate does not necessarily constitute a total failure of your security strategy. And each of these standards will also take you through a process of assessing what do you have, where would you like to go with your system?

There are things that you should always do, like identifying and training stakeholders, taking inventory of your networks and devices, developing clear policy, communicating that policy. Usage of more practical things like usage of DMZs between OT and it networks, those types of things that take you through your security life cycle. 

There's also a good deal of flexibility, though, in how you apply certain controls represented in these standards. You don't have to mitigate every conceivable event. You don't have to apply every single control that's that's available to you. In fact, you probably shouldn't. You have to, you may have a PLC on the backside of your facility that has very little impact.

If it's attacked in isolation. And it's in those cases, it's perfectly fine to say, let them have it if they want it. If there's, if that's where your appetite for risk is as a business so you have to take these standards, digest them and understand them well enough to apply them and evaluate them against your appetite for risk and your your business case for that how security or not having security tied to a certain device or process or area. 

So with that flexibility, then there come a lot of questions that inform decisions what do we mitigate and how do we design networks to suit our particular need while also considering security. What should we be backing up? How should we back up? How often should we back it up? How should you manage security patches on machinery that had high availability requirements? How do we supply or apply security heavily enough to be useful, but lightly enough as to not stifle support or production, because if you're overbearing with your security policies such that it affects production and affects support, then people start looking for ways around it.

Chris: 17:02 

Trying to break the system. 

Richie: 17:04 

Yeah. And people find ways you might have a, an excellent badging system that is stifling to people who want to run machinery. And invariably, it'll come down to someone sharing badges and just to keep the line running because the thing that is most in people's faces is production and numbers and things like that. So you wanna, you want to moderate your security in a way that isn't stifling to those people and communicate the importance of the measures that do exist. So that everybody has a common understanding of their role and in terms of security.

And so these types of topics and this type of answering these questions. You know, you might benefit from consulting a company that has hands on experience in securing the systems and the practical and reasonable way. Prism systems for whom I work, we were one of those companies that has proficiency in the OT space. There are also knowledgeable people working at the manufacturers sorts of the major manufacturers for controls products that can offer general guidance and security, as well as guidance specific to securing their own devices. 

Chris: 18:05 

You took me right to where I was going to go. Where I may be in a plant and I don't have that, that personnel who has the understanding like you do, obviously all around these topics, but I want to get there. So as it really teaming up with a consultant like yourself, or like you said, the manufacturer partners that build these systems, that's a really good step you recommend?

Richie: 18:26 

Yeah. I mean, I think you can apply the same concept that's behind defense in depth. Don't limit yourself to a certain layer, read the standards yourself try to understand them at least conversationally. And yes if you find yourself struggling with or questioning some of the decisions that are being made about security policy, you should always reach out to someone who's done it before.

That's not different from any other discipline in life. And do lean on the experts. You know, there's a lot of attention going towards industrial cyber cybersecurity. And that's no better reflected in what you'll find at the major manufacturers they're thinking about these things. They have talented and smart people who are interrogating security of their own devices and can certainly be helpful. 

Chris: 19:08 

No doubt and sometimes the industrial environment just has a different mindset. So any misconceptions out there around safety and security that you like to just expose or talk about? You mentioned several already, but just curious. Is there anything that's missing from the industrial standpoint you'd like to talk about? 

Richie: 19:31 

So aside from the notion of safety as existing, entirely separate from security that we've already talked about I've heard the term and I like it a security by optimism. It's a well-represented misconception in industry. And what I mean by that is that you hear concepts and comments like why would anybody target our facility? If you're making widgets for some niche industry or something like that, you might be a hundred percent correct in your optimism that you're not ever going to be targeted.

It's hard to get into the minds of some of these people who execute these attacks. And the fact is. Increasingly accessible attack toolkits that remove most of the demand for technical ability on the part of the attackers, there are YouTube videos and tutorials that can take a fairly non-technical person through all of the steps required to use these readily available attack tools that can be just downloaded by anyone.

And you know, the idea of a hacker is stylized and our entertainment. And there are people who execute attacks just for the image of it. Just to feel cool about it, but most facilities who think that they won't be a target of a sophisticated or state sponsored group are probably correct.

And this is obviously not true for critical infrastructure and things like that, but like we said before, the many attacks that affect OT system It's a result of these common protocols and there it's really just collateral attacks, collateral damage, nobody targeted a specific facility.

You do have targeted facilities, but but a lot of these attacks are just incidental. And in fact, the computer got plugged in to the network somewhere to troubleshoot some problem. And then weeks later, the system starts acting funny and you start losing resources and the system goes down.

Now, so whether or not a facility would be targeted or is likely to be targeted, it's a bit of a straw man. And the degree to which you're exposed and the level at which you're prepared to defend and detect and remediate cybersecurity events those are more useful metrics. They're more useful than when discussing security then a metric like such as the probability of a targeted attack because exposure does not, it doesn't depend on whether you're targeted or not. It's just a thing that happens. 

So you also hear comments like we've been fine thus far. This has been, and this goes back to S you know, my comment on why are we still fighting this in industry? It's the, the root of the complacency and the sluggishness to adapt and industry.

So it's you hear comments like, we've been fine, so why prioritize security now? And this is again, an overly optimistic approach. Those who are involved in designing safety systems should see an analog here, in safety it's often said that just because a hazardous event hasn't happened yet doesn't mean that it won't happen over the lifecycle of a system.

So there's a similar process in a similar view between safety and security. The problem is a lot of places still aren't taking the security side of things seriously enough to do an earnest risk assessment within the context of security. So we should approach the two topics with similar respect.

And I'll say again, if the system's not secure, it isn't safe. So if we're serious about safety we need to be more serious and more earnest about our assessment and our treatment of security issues. 

Chris: 22:44 

No doubt. I love it. There were so many things you unpacked there around common misconceptions and Richie, you're definitely such an expert in this. I can't thank you enough for what you've shared with our listeners today. A very enlightening for me. I know I just sat back and listened. Took a lot of notes, maybe tied together the why. We always wrap up with the why on EECO Asks Why, so why is having that clear understanding of safety and security so important for those industrial end users out there?

Richie: 23:12 

Well, we're hearing more and more about attacks on control systems and it's not that we're any worse at securing these control systems today than we used to be. And it's only partially due to the fact that attackers are more aware of control systems and more inspired to attack them as targets. But we've been passively protected in the past by the protocols like device that my boss PROFIBUS there's types of protocols that aren't built on the TCP IP backbone and are thus outside of the tool kit and the comfort level of most attackers. Now the OT communications are moving more and more onto TCP IP based protocols.

These protocols have vulnerabilities that are well-known to attackers who are specifically targeting systems. And there are toolkits built around these specific vulnerabilities. So aside from the targeted attacks and probably more dangerous widespread usage of these protocols and the OT environment expands the window by which these non-targeted attacks might make their way onto our OT network. 

And a big example. Yeah, a topical example would be ransomware attacks. So ransomware attacks are designed to be highly infectious and they're not typically rigorously targeted attacks. The business model of a ransomware attack is the more devices I affect with my attack, the more profitable that becomes for me. So I need to write this thing such that it infects everything. So you're looking for widespread coverage of you're tech and there other things incentivizing ransomware attacks. And then if you're already in the system, why not trigger a ransomware attack? It's another way to monetize your ill intent. 

But because there are these highly infectious wide coverage pieces of malware out there. Many OT, cybersecurity incidents are not actually targeted attacks, but they're are one of these more general, highly infectious attacks that make their way onto an OT network and then cause a ton of collateral damage.

This is all made possible and I've referenced it before about due to the widespread adoption and usage of TCP IP based protocol. All that said I'm not advocating for abandoning these protocols at all. They're very useful. They're easy to work with. They offer a wider opportunity for connectivity which is a good thing if treated correctly. These are all good things. We just need to be better about how we design and secure our networks. We can't rely on security by optimism or security, obscurity any longer because we're now moving into these areas where we're operating on a well-known grounds. 

Chris: 25:40 

For sure. Well, Richie, this is again, so much wonderful information for the listeners out there that want to learn more. Maybe even see that demo, check out the show notes. We'll have the information to get connected with Richie directly to ask questions and thank you so much for all the information you shared. So i  was a scary topic, to be honest, when you really sit back and think about it, but if you take the good approach, like you've really laid out for our listeners here we can all be more safe and more secure. So thank you sir.

Richie: 26:08 

Thank you. 

Chris: 26:11 

Thank you for listening to EECO Asks Why this show is supported ad free by Electrical Equipment Company. EECO is redefining the expectations of an electrical distributor by placing people and ideas before products, please subscribe and share with your colleagues and friends. Also leave comments, feedback, any new topics that you would like to hear. To learn more or to share your insights, visit EECOAsksWhy.com. That's E E C O A S K S W H Y.com.