The issue of lacking OT operators and security professionals

30 November is the computer security day. Computer security can only be ensured if one is aware of its mistakes and shortcomings. The purpose of this blogpost is to highlight the importance of lacking OT (operational technology) operator issue. The main question is: will the knowledge required for OT security be available in the future?

In the last decade Black Cell audited various Hungarian designated critical infrastructures, and our team found out that in general, the biggest problem of OT operators is that the stakeholders with special knowledge are single point of failures within the organization’s security establishment. The OT and security related knowledge that accumulated over several decades in the organizations, especially in terms of the legacy industrial control systems, are specific in each organization. When the auditor asks why one is the only one familiar with these tasks, the answer is always “this is how it has evolved over time”. These professionals have been working in the same role at the same organization for years, and their knowledge base is invaluable. Yet, when these reliable employees will retire with their valuable knowledge, their expertise and knowledge must be retained within the organization and passed over to other stakeholders.

This situation is natural, however it generates a risk in the business continuity and disaster recovery, because only these professionals are aware of the specialties of the OT operation. Furthermore, it is a common case, that these experts have no replacement and there is no handbook or other relevant guideline that contains all the expertise and collected know-how. When a process is functioning, the management often is inclined to think that everything is all right, however, forgets that this is a volatile state. We frequently identify this risk the IT/OT security audits, but unfortunately it is rarely being mitigated. There is another regular issue: in the past many persons have worked at only one or possibly two workplaces in their entire career. There was very small risk of fluctuation in the provision of highly trained and skilled human resources. Nowadays, employees often change jobs to increase their salaries, receive a wider set of benefits or enjoy flexible working conditions. As a result, they do not have the opportunity or time to transfer the knowledge or gain the practice and expertise neither in operating nor securing individual industrial control systems, because by the time the new employees would have gained a deep understanding, they moved on to a new job.

The risk of failure rapidly increases if the OT operators and information security employees retire, and juniors don’t have the necessary guidance in written form, might provided by OT system special practical training. In those organizations where we uncovered such risks five or ten years ago, there was enough time to come up with and implement the right problem-solving plan. In other cases, where the risk was not identified yet, the management will be facing serious issues. The worst-case scenario is that the single point of failure OT operators and security professionals are going to be retired, and the knowledge would also be lost. Here are some good practices to avoid this risk and treat the root cause:

• Develop a strategy to ensure retain or train employees who possess the required special knowledge.
• Hire engineers a reasonable time before the senior professionals retire, giving them enough time to learn from the OT professionals and make them capable of applying the knowledge.
• There are industrial control system operators where the “veterans” are about to retire, so they have documented the specific workflows and solutions to problems that frequently arise. This way the guidance will be available for the next generation of OT operators and security workers.
• Another example of a good solution is to contract with universities and other educational institutions to offer scholarships to trainees for completing an exercise in the critical infrastructure and OT operator organizations. During the scholarship period, trainees can learn the special OT operation and security tasks besides later can be employed by that company.
• Rethinking organizational processes is also a way of dissolving previously created single point of failure jobs: more colleagues need to be involved in the processes. Other solution is to replace legacy systems, however a more expensive solution, which results the old knowledge and process to become insignificant. It is much more complicated than it sounds. Such a project can take several years and without an appropriate ROI, decision makers would not consent to it.

The risk mitigation isn’t possible without people and learning management experts, CFOs, COOs, CIOs, and senior managers who are interested in this issue.

There are many trainings where OT operator and security knowledge can be gained. Black Cell recommends many ICS/OT relevant trainings in its monthly ICS security feed, available our website. As we mentioned earlier, every industrial control system is different, and the security solutions and the organizational specialties also differ. The educational background itself will most probably not enough to handle these specialties.

It makes the situation worse, that many IT security professionals are missing from the IT sector. Digitalization is constantly evolving, and the number of professionals is not growing to parallel extent. In many cases, the availability shortage of IT security professionals affects industrial control system operation as well, since usually there is a close interaction between OT and IT.

So, the key message is develop a strategy to address the issue by having processes documented, skilled employees retained and new talent continuously involved to prevent knowledge to disappear.

If your organization have OT we strongly recommend to assess the relevant risks. Black Cell created a tool, which can help to assess the OT professional relating risks. You can reach the tool here:

Zsolt Baranya, 11.30.2022.