Recovery Starts with Better Change Management

Before and after a cyber-attack, robust change management techniques can ensure production uptime and resilience. This feature originally appeared in AUTOMATION 2022 Volume 4: Cybersecurity & Connectivity.

Cyber risks are on the rise. Manufacturing and industrial operations long ago shed their belief that they were invulnerable because their systems were too isolated or too obscure to be targeted. Organizations now seek to understand the impact and likelihood of their cybersecurity risks and then seek to reduce those risks. But mitigating risk is only the first part of comprehensive cybersecurity planning.To enable continuous operations and limit business impact when a cyber-attack does occur, organizations need additional tactics. These can include strategically hiring cybersecurity talent, or using new methods to identify, combat and recover from attacks. Given increasingly interconnected and frequently updated systems, robust software-system management is a particularly useful tool.

Improvements to a plant's operational technology (OT) cyber risk management and mitigation plans should be made in tandem with the rise of attacks, which are up 144% from 2020, according to Industrial Safety & Security Source (ISSSource). Data from the company's OT Security Incidents in 2021: Trends & Analyses report, which analyzes data from its ICSSTRIVE.com database, says "the year 2021 saw the number of cyber-attacks with physical consequences in process and discrete manufacturing industries more than double over those reported in 2020…. Almost all these incidents were the result of targeted ransomware. Almost all these attacks impacted multiple physical sites."In a recent webinar, ISSSource reported that OT ransomware incidents with physical consequences have increased 133% year-overyear since 2020, and that published estimates cite up to $140 million in damage per event. The types of facilities subject to these attacks cover the industry spectrums (Figure 1).

Automation can be used to counter more of the sophisticated attacks coming at organizations, said McKinsey. "Automation should focus on defensive capabilities like security operations center (SOC) countermeasures and labor-intensive activities, such as identity and access management (IAM) and reporting. AI and machine learning should be used to stay abreast of changing attack patterns. Finally, the development of both automated technical and automatic organizational responses to ransomware threats helps mitigate risk in the event of an attack."As the level of digitization accelerates, organizations can use automation to handle lower-risk and rote processes, freeing up resources for higher-value activities, McKinsey advises. Automation decisions should be based on risk assessments and segmentation to ensure that additional vulnerabilities are not created. For example, organizations can apply automated patching, configuration and software upgrades to lowrisk assets but use more direct oversight for higher-risk ones.As ransomware attacks increase, organizations must respond with technical and operational changes, adds McKinsey. "The technical changes include using resilient data repositories and infrastructure, automated responses to malicious encryption, and advanced multifactor authentication to limit the potential impact of an attack, as well as continually addressing cyber hygiene. The organizational changes include conducting tabletop exercises, developing detailed and multidimensional playbooks, and preparing for all options and contingencies—including executive response decisions—to make the business response automatic," the report states.

Regardless of their sector, size, or task set, industrial production environments require complex information technology (IT) setups designed to handle integrated systems and high volumes of data. While both OT and IT departments use digitalization to improve productivity and other business outcomes, sometimes the two worlds require translation and teamwork so their different methods and best practices can be achieved. For example, although it has become standard practice for IT departments to schedule routine backups and manage data storage, OT personnel have been slow to adopt such data hygiene and software-system management solutions.Ensuring that the correct, authorized versions of software are always running is paramount to keeping production running—whether OT or IT personnel are tasked with managing the system. With version control and change management tools, operators can have access to the most current software and know when changes require further action. Advanced software solutions can summarize the entirety of an automated production environment and analyze devices on the shop floor. Because they can detect differences in programming configuration and firmware versions, even for identical sensors, such systems make it easier to isolate errors.Change management is a structured process for planning and implementing new ways of operating. According to an article in the April 2022 issue of InTech, the official publication of ISA—International Society of Automation, "An automated, standards-based documentation process saves time and cost while increasing quality. With standardsdriven processes and workflows comes the assurance of following the best industry practices during the definition, design, development, integration, documentation and support of automation projects. This ensures the execution of projects with precision and standardization."Successful change management relies on four core principles:

That's just the beginning. If changes are made by multiple people to OT-centric code, the potential exists for one group to not know what another group is doing. If changes that affect the operation—and/ or cybersecurity—of a manufacturing facility are made, they must be made for a valid reason. There must be documented justification for the change. If a robust change-management system is in place, that system should track and catch any deviation from what is expected in the procedures or code. The system's activity history will reveal who changed what, where, when and why.Good change management is not a one and done proposition, nor is it best executed by spot checking. Good change management must be in place continually—in real time. Whether unauthorized changes come from lack of employee communication, unauthorized OT system users, or actual cyber malfeasance, change management that's done correctly is a company's best insurance against downtime and resource for recovery.

Version control and software change-management tools can help organizations at all stages of cybersecurity activity—from detection of vulnerabilities to recovery from attack. They can be used to ensure data is current and corresponds to the latest iteration. They can reveal data anomalies and, hence, vulnerabilities and exposures.A state-of-the-art change management system can manage software programs and configuration settings data in a standardized way, so change history can reveal who changed what, where, when and why. Such tools can aid the user in managing insecure protocols, misconfigurations and other vulnerable security points, as well as provide automatic assessment of vulnerabilities, affected assets and the entire industrial control system. Through threat detection functionality, the tool can automatically discover, protect and manage an industrial control system's critical assets and provide users with risk and vulnerability reporting.When a production system malfunctions for whatever reason, maintenance staff can take an average of three or four hours to track down changes using a manual approach to managing software versions. Automatic backups reduce the downtime and facilitate rapid recovery. The backups enable users to restore the last authorized version or an earlier one if that was the one running before the malfunction occurred.When version control and software change-management systems are installed on premises, on the OT network side of the factory floor, tasks from detection to backup to recovery can be automated. The system should provide for the IP addresses of assets and devices to be scanned and added to the system even if they are located below the programmable logic controller (PLC) level. The information gathered should include the device brand, manufacturer, firmware version and where the device is physically located in the rack.When users run an automated backup, it should also be possible to send this information to a threat analysis component, which can check the web to see if there are vulnerabilities in software components or firmware versions. It is also helpful if the threat analysis can identify unusual traffic patterns, malware, or external, unapproved access to the network.

An example of a state-of-the-art change-management system is octoplant from AUVESY-MDT. octoplant is a new data management platform that provides a vendor-independent and comprehensive view of all automation backup processes involving OT and IT. This changemanagement platform consists of eight solution sets tailored to specific industrial needs (Figure 2).

Cyber-attacks are on the rise so improvements to cyber risk management and mitigation plans should be made now. With increasingly interconnected systems and exponentially growing quantities of data, industrial operations must put systems in place that allow them to detect vulnerabilities and respond when a breach occurs. Robust data and software-system change-management tools can help ensure companies remain running during normal operations and respond quickly when the worst occurs and prevent production downtime during normal operations and ensure resilient recovery when the worst occurs.This feature originally appeared in AUTOMATION 2022 Volume 4: Cybersecurity & Connectivity.

Jack Smith ([email protected]) is a contributing editor for Automation.com and ISA's InTech magazine. He spent more than 20 years working in industry—from electrical power generation to instrumentation and control, to automation, and from electronic communications to computers—and has been a trade journalist for 22 years.

Check out our free e-newsletters to read more great articles..