4
Industry and Operational Technology/Control System Security

AN INDUSTRY PERSPECTIVE ON OPERATION TECHNOLOGY CYBERSECURITY

Chuck Weissenborn, public-sector chief technology officer at Dragos, said cybersecurity has focused on confidentiality, integrity, and availability, but as the discussion in the last session noted, cybersecurity should include human safety. NIST owns this, but it is up to the workshop participants and practitioners in the field to drive that effort. “We all have a responsibility to our people and our missions to do so,” he said.

Industry, said Weissenborn, cares about cybersecurity because there are at least 24 threat activity groups focused on attacking OT, with more appearing on the horizon. Most probably operate at the nation-state level, and they have demonstrated that they want to position themselves to affect or have affected an OT system. “Some folks will say we are waiting on a cyber Pearl Harbor,” said Weissenborn. “I would argue that in this space, we have already seen that. It has already happened, again, multiple times, both associated with conflict and with competition.”

For example, VOLTZITE is an aspect of Volt Typhoon focused on OT and critical infrastructure systems. “These are the guys that have taken that initial access that is being provided and actually dove down into environments, working to place themselves lower and lower and closer and closer to the systems that actually operate and have an element of physics aligned to them,” said Weissenborn. These actors are sophisticated and change their tactics, techniques, and procedures as they move through environments to evade detection. “It is hard also to frame the problem of VOLTZITE and Volt Typhoon because we do not know how deep or how wide their penetration actually is,” he added.

Another example is KOSTOVITE, which compromised an electrical generation facility in Australia through a virtual private network vulnerability. What is interesting about KOSTOVITE, said Weissenborn, is that it did no reconnaissance, but went straight from initial entry to the control panels of the 30-plus MW generation facility. This attack then used vulnerabilities in one vendor’s network storage devices to jump to a large-generation facility in Arizona. “KOSTOVITE reached the point where they had the ability to cause a direct, physical impact to OT systems. They were on the controls for the generation capabilities. They could have disabled safeties. They could have turned things on and off,” said Weissenborn.

He said the first question his company’s commercial customers ask is whether a threat actor has compromised their environment, a question arising because they have no visibility of what is moving across their systems. Yes, some of these companies have endpoint detection and response solutions, but programmable logic circuits do not support such solutions, and adversaries are using programmable logic circuits as part of their command-and-control network within OT environments. “Relying on the ability to see something on an endpoint is an outdated approach, unfortunately, for OT now,” said Weissenborn.

Next, customers are concerned about which dependencies of their critical functions and assets should concern them, a challenging question to answer until there is visibility about how an asset is operating on a daily basis. They also want to know if their systems are operating as expected and if a vulnerability is something they need to address. As Weissenborn explained, an OT system may have a vulnerability in a

legacy system that would require massive investment to address. Finally, customers want to know if their OT cybersecurity program is effective. Most questions, he added, focus on safety, reliability, and productivity rather than confidentiality and integrity. “In industrial organizations, safety is the number one consideration above all else,” he explained. “One of the reasons why OT systems exist in the first place is because they were trying to make things more safe.” Additional drivers in the commercial space include insurance requirements for cybersecurity riders, meeting government regulations, competing for government contracts, and profits. Weissenborn noted that commercial firms will not modify contracts for free when the modification calls for something above and beyond the industry standard.

The industrial CS and OT cybersecurity journey starts with assessing, planning, and organizing. This includes establishing an incident response plan and gaining an understanding of the architecture of a firm’s systems and an inventory of the system’s components, perhaps using technology the firm will deploy later and implementing multifactorial authentication for any remote access or other systems. He noted that centralizing OT for cybersecurity is difficult in terms of execution and knowledge, which is why he always encourages companies to have people at the site level who understand the system and can do the basics of cybersecurity.

Once these baseline actions are complete, said Weissenborn, the next step is to operationalize OT cybersecurity controls. This is where the company would deploy continuous monitoring at key locations for identified vulnerabilities. It is also when companies start to gain the ability to detect and respond to OT incidents on their own and to understand and identify interconnections that may be vulnerable. However, he added, understanding how to do this analysis is challenging for people who do not do it regularly.

With the OT system operationalized, the third step on the cybersecurity journey is to optimize the system and develop a risk reduction program. The federal government has programs such as DoD’s Mission Assurance program, and there are commercial partners with expertise in certain domains. “Just make sure when you are engaging folks that you are asking the right questions and not trying to force somebody to work outside the domain they are an expert in,” said Weissenborn. For example, an expert in electric grid systems would not be the person to ask for advice about a manufacturing system. This entire process, from baseline activities to optimization, can take from 3 months to 3 years.

Weissenborn listed several key capabilities and resources that industry uses for cybersecurity. Security information and event management technology can be useful, but he said to exercise caution in terms of structuring and managing data when combining that technology for IT and OT systems. Data historians, another useful capability, are repositories of process-related data and are one of the first sources of information following an incident. Systems for detecting configuration changes go hand in hand with threat detection, and there are now systems that look at percentage increases in central processing unit use, which can provide great contextual information to support a cybersecurity program. Endpoint security logging is another useful capability the IT community uses routinely, but this is less common in the OT area.

In cybersecurity, people are the number one problem. One challenge is that training for cybersecurity professionals aims to meet one standard and create generalists. “I do want not a generalist in OT,” said Weissenborn. “I do not want a generalist who has never opened up an electrified cabinet or a cabinet inside of an electrified substation. I want somebody who actually knows how that stuff works, how to use that disconnect that is on there that may not be present in another space,” he said. He added that not enough networking people understand OT and how it is different from IT, which is a problem given how adversaries have perfected the art of compromising telecommunication and network infrastructure devices. “This is going to be a problem in the next year or two,” said Weissenborn. IT systems also exist with OT, which means if a workstation can program a device or signal a piece of equipment, it should not be used to surf the internet.

Weissenborn said companies need to have knowledgeable people at all levels and locations and not just at a central location when it comes to the cybersecurity of OT. One reason is that someone needs to get to a compromised component promptly. Locally based people can also participate in cyber

commissioning or site acceptance testing, which is the process of looking at a system and saying it is ready to be put into use. At the enterprise level, it is possible to centralize a company’s understanding of operational and mission risks and perform certain cybersecurity service provider functions and security control validation.

At the strategic level, having people who understand OT can help with resourcing decisions. They can also translate OT cybersecurity and engineering to senior leaders in a way that resonates with them. Also needed, said Weissenborn, is well-defined OT cybersecurity governance that includes someone designated to make decisions.

Weissenborn concluded his presentation with the five critical controls for effective OT cybersecurity:

• An industrial CS-specific response plan
• A defensible architecture
• OT visibility, including an asset inventory, vulnerability mapping, and monitoring
• A vulnerability management program
• Multi-factor authentication

Doing these five things, he said, will address 90 percent of the threat actor scenarios and the tactics, techniques, and procedures they are using today. He noted that anyone who is an OT operator or owner can sign up for his company’s OT-Cyber Emergency Readiness Team (OT-CERT) program dedicated to addressing the OT resources gaps that exist in the nation’s industrial infrastructure. OT-CERT is a community of OT asset owners and operators that works at no cost with Dragos team members on how to accomplish best practices and provide structures to help get people started on OT cybersecurity. OT-CERT has 1,600 members in 60 countries. “This is something that we put together because people were getting the ‘what to do’ but not the ‘how do we actually do this if we are X type of organization,’” said Weissenborn.

Dragos also provides OT cybersecurity software technology for no charge to small water, electric, and natural gas providers with under $100 million in annual revenues. The goal is to help reduce the risk of cybersecurity events by working with these small firms to inventory their assets, detect and hunt threats, manage vulnerabilities, and respond to incidents. The only requirement is that the company becomes part of Dragos’s information intelligence–sharing platform that helps generate trends data and other information that CISA and the National Security Agency are using. Sandy Kline noted that DoD cannot accept free services.

Weissenborn noted that the Navy is the first defense organization to put a cybersecurity monitoring requirement in all design and engineering plans. He hopes this becomes the standard across DoD, if not all industry. One advantage of doing this in the military is that it enables using military construction funding instead of relying on cybersecurity funding, including for software maintenance subscriptions.

TECHNOLOGY CONVERGENCE, SILOS, AND INVENTORY

Coby Jones, senior manager of advanced applications for Johnson Controls Federal Systems, said that commercial demand from banking, telecommunication, cloud services, and real estate drives the CS and OT industry and the features it puts into its products 90 percent of the time. Cybersecurity is another layer that industry demands on top of CS and OT. He commented on the importance of having people in place who understand the interrelationship of different systems, which requires cross-function training. He also noted that industry has engineered safety into its designs, but what is often missing is programming for when a cyberattack sends false information indicating everything is okay when it is not. Jones added that nobody in the CS and OT systems industry is pursuing participating in the FedRAMP because it is

expensive to go through FedRAMP and expensive to maintain it for years afterward. Jones said, though, that the controls industry is looking at incorporating cybersecurity into its products.

Turning to the subject of technology convergence, Jones said that IT systems are used for data-centric computing, while OT systems monitor events, processes, and devices. Convergence is the ability to grapple with both worlds and combine the capability of computers, electromechanical devices, and manufacturing systems and capitalize on the massive amount of data available. Today, operators of OT systems and an organization’s IT team are coming at the same problem from different places and often speak different “languages,” creating silos. He said that companies know how to use available tools to scan their IT networks, but the first company that invents a tool that can scan every OT system will dominate the market, although this is not likely to happen in his lifetime. “There are too many systems out there to get the real data you want to get out of each system,” said Jones. “It is going to take partnerships.” Another challenge is that OT integrators prioritize function and operation over security and assign administrative privileges to people who might not need them.

Referring to the five levels in the Purdue model for industrial CS security mentioned in the workshop’s first session, Jones said that today, Level 0 is probably not relevant, although it may be when everything is IP-based.⁶ However, vendors should be able to provide some details about their system, how many users are on the system, and how many have administrative privileges or who do not have passwords that expire. “If you do not know that, then knowing your inventory does not really matter much,” said Jones. “You have to know what is inside your system and outside your system.”

DoD, said Jones, does not have the money to replace all its systems or even the critical infrastructure systems. There will need to be a way to prioritize what gets replaced. He advised talking to vendors to find out which of their controllers need updating using the inventory analysis. This information will enable creating a roadmap and diving deep to understand where vulnerabilities lie within firmware and the software running on a system. While funding is an issue, it is time for installations to start asking for funding now, said Jones, and that will require knowing exactly what they need to upgrade. He said that DoD should emphasize a cross-functional team approach that includes IT, operations, safety, and management and to develop a plan to get an accurate inventory list, preferably using an automated system. He also recommended creating and performing cross-training activities and asking facility-related CS vendors for support.

GUARDING AGAINST OPERATION TECHNOLOGY SUPPLY CHAIN–BASED ATTACKS

In the workshop’s final presentation, Robert Hunter, founder of Alpha Guardian, said that cyberattacks use OT and IoT as major sources of data breaches. “It is critical to understand that, at the end of the day, the bad guys are looking at OT devices because in most cases, they are using very old, antiquated protocols,” said Hunter. Even though some of those protocols have been updated, the new protocols are not backward compatible with the original protocols. He noted that even one device with an old, vulnerable protocol mixed among a hundred more secure devices renders a network vulnerable. He also pointed out that CISA and DOE warned in 2022 that threat actors are even gaining access to Internet-connected uninterruptable power supply (UPS) devices, which is concerning, given that these devices sit in close proximity, both virtually and physically, to IT systems. In fact, UPS devices had been ranked as the most vulnerable targets among OT systems until programmable logic controllers (PLCs) took over the number-one spot (Ribeiro 2024). PLCs, said Hunter, are perfect targets for cyberattacks because they were never designed with cybersecurity in mind (Schaefer 2023). In addition, the primary protocols they use are mostly older, vulnerable ones.

___________________

⁶ More information about the Purdue model can be found in Garton (2019). Level 0 comprises the actual physical processes that CS control.

Hunter explained that if malicious actors can gain entry to any OT system, they can ultimately move into the IT system and access information. “One of the reasons for this, which I do not think is terribly well understood, is that IT and OT systems do share common protocols,” he said. One example is the Simple Network Management Protocol, which manages almost every server and network switch. While there is segmentation that makes it more difficult to move from one segment of the network to another, an attacker can still succeed given enough time in the network. “Especially when it comes to government, this is critical because we have enemies of our state and those enemies will stop at nothing and they have well-funded budgets to go after that information,” said Hunter.

As an example, Hunter cited the case where hackers successfully attacked a casino through an Internet-connected fish tank thermometer. Although it took the attackers “quite a bit of time,” they eventually got into the casino’s finance system and stole a “massive amount of money.”

Supply chain–based attacks are increasing and have become common, said Hunter, with 91 percent of all organizations experiencing a software supply chain attack in 2023 (Security Staff 2021). These attacks use a manufacturer’s software or firmware update process to inject malicious code into a device or exploit a vulnerability with that updated vendor code. In addition, the number of firmware vulnerabilities has skyrocketed over the past 3 years. Security researchers, said Hunter, believe the number of common vulnerabilities and exposures (CVEs) is 7.5 times greater than what was documented 3 years ago (Epp 2024), leading to a nearly tripling of exploitations in 2023 (Jones 2024). Hunter noted that just because CISA and NIST have listed a CVE does not mean that CVE is the only thing that could be wrong with a device. His firm, for example, has found several devices and specific code flaws that other researchers had not yet identified. His team has not published this information because of the alarm it might trigger, but they are taking specific action in the marketplace.

Hunter said the Internet of Things Cybersecurity Improvement Act of 2020, after languishing in Congress for 2 years, was enacted 2 weeks after the SolarWinds supply chain attack was made public. This law focuses on smaller IoT devices such as temperature sensors and security cameras. The subsequent May 2021 Executive Order, EO14028, specifically called out OT cybersecurity as a requirement for all government facilities.

The SolarWinds attack, said Hunter, is still the largest published attack on government sites. It was a perfect attack, he said, because it went after a network management system that the U.S. government and government contractors used heavily. The attack occurred in such a way that SolarWinds had no idea their servers had been attacked. Here, the Russian attackers took the SolarWinds code, disassembled it, added malware, and hid it by making sure the altered code had the exact same number of bits as the unaltered software, hiding it from the company. Once the malicious code was back in the SolarWinds system, SolarWinds began sending out software updates with the embedded malware to steal valuable government data. Some of these data were used in 2024 to attack a large Internet service provider in the U.S. South.

Another way such attacks have occurred is through various domain name systems, said Hunter. In one case, hackers caused a legitimate server request aimed at a vendor software update server to be redirected to a malicious software server. This approach targeted a large industrial CS-OT manufacturer, and Hunter and his colleagues believe there were probably tens of thousands of software copies infected with malware distributed before the attack was discovered. Similar attacks have hit PLC firmware updates.

To guard against supply chain attacks, Hunter recommended employing the zero trust method of OT security, which means not trusting a vendor’s software, firmware, or employees. Implementing zero trust requires these actions:

• Isolate and segment all OT systems from IT systems.
• Use a threat discovery and response solution that the government has tested to continuously scan the OT network for signs of malicious activity.

• Use a firewall designed specifically for OT systems, which can include a data diode or traditional firewall that has been tested on government sites.
• Never allow a vendor to access the OT network without permission and surveillance, and do not accept the excuse that failure to allow access to the network “as needed” will violate the equipment’s warranty.

QUESTION AND ANSWER SESSION

A workshop participant asked if there is a need to check every controller on a system. Jones replied yes, adding that a good retro-commissioning should create a detailed inventory of existing systems. What an inventory will not provide, though, is information about device firmware and if it needs updating.

An online participant asked Jones if his company was concerned about supply chain risk management, given the number of devices that have chips and circuit boards manufactured and assembled overseas. Jones said his company is “very conscious” of that and is constantly assessing its suppliers and their ability to meet various security requirements. He noted that ISASecure requirements should probably be incorporated into the NIST requirements and the UFC.

This page intentionally left blank.