This paper demonstrates the potential for autonomous cyber defence to be
applied on industrial control systems and provides a baseline environment to
further explore Multi-Agent Reinforcement Learning's (MARL) application to this
problem domain. It introduces a simulation environment, IPMSRL, of a generic
Integrated Platform Management System (IPMS) and explores the use of MARL for
autonomous cyber defence decision-making on generic maritime based IPMS
Operational Technology (OT). OT cyber defensive actions are less mature than
they are for Enterprise IT. This is due to the relatively brittle nature of OT
infrastructure originating from the use of legacy systems, design-time
engineering assumptions, and lack of full-scale modern security controls. There
are many obstacles to be tackled across the cyber landscape due to continually
increasing cyber-attack sophistication and the limitations of traditional
IT-centric cyber defence solutions. Traditional IT controls are rarely deployed
on OT infrastructure, and where they are, some threats aren't fully addressed.
In our experiments, a shared critic implementation of Multi Agent Proximal
Policy Optimisation (MAPPO) outperformed Independent Proximal Policy
Optimisation (IPPO). MAPPO reached an optimal policy (episode outcome mean of
1) after 800K timesteps, whereas IPPO was only able to reach an episode outcome
mean of 0.966 after one million timesteps. Hyperparameter tuning greatly
improved training performance. Across one million timesteps the tuned
hyperparameters reached an optimal policy whereas the default hyperparameters
only managed to win sporadically, with most simulations resulting in a draw. We
tested a real-world constraint, attack detection alert success, and found that
when alert success probability is reduced to 0.75 or 0.9, the MARL defenders
were still able to win in over 97.5% or 99.5% of episodes, respectively.

该论文展示了自主网络防御在工业控制系统中的潜力，并提供了一个基线环境，进一步探索多智能体强化学习在此问题领域的应用。

多智能体强化学习在海上操作技术网络安全中的应用

Multi-Agent Reinforcement Learning for Maritime Operational Technology  Cyber Security

Industry 4.0 is driven by demands like shorter time-to-market, mass
customization of products, and batch size one production. Reinforcement
Learning (RL), a machine learning paradigm shown to possess a great potential
in improving and surpassing human level performance in numerous complex tasks,
allows coping with the mentioned demands. In this paper, we present an OPC UA
based Operational Technology (OT)-aware RL architecture, which extends the
standard RL setting, combining it with the setting of digital twins. Moreover,
we define an OPC UA information model allowing for a generalized plug-and-play
like approach for exchanging the RL agent used. In conclusion, we demonstrate
and evaluate the architecture, by creating a proof of concept. By means of
solving a toy example, we show that this architecture can be used to determine
the optimal policy using a real control system.

本论文介绍了一个基于 OPC UA 的运营技术意识强化学习体系结构，扩展了标准的 RL 设置，将其与数字孪生的设置相结合，与此同时，定义了一个 OPC UA 信息模型，允许进行通用的即插即用式交换 RL agent。通过解决玩具示例，本文演示和评估了体系结构，证明了该结构可用于使用真实控制系统确定最优策略。