This paper demonstrates the potential for autonomous cyber defence to be
applied on industrial control systems and provides a baseline environment to
further explore Multi-Agent Reinforcement Learning's (MARL) application to this
problem domain. It introduces a simulation environment, IPMSRL, of a generic
Integrated Platform Management System (IPMS) and explores the use of MARL for
autonomous cyber defence decision-making on generic maritime based IPMS
Operational Technology (OT). OT cyber defensive actions are less mature than
they are for Enterprise IT. This is due to the relatively brittle nature of OT
infrastructure originating from the use of legacy systems, design-time
engineering assumptions, and lack of full-scale modern security controls. There
are many obstacles to be tackled across the cyber landscape due to continually
increasing cyber-attack sophistication and the limitations of traditional
IT-centric cyber defence solutions. Traditional IT controls are rarely deployed
on OT infrastructure, and where they are, some threats aren't fully addressed.
In our experiments, a shared critic implementation of Multi Agent Proximal
Policy Optimisation (MAPPO) outperformed Independent Proximal Policy
Optimisation (IPPO). MAPPO reached an optimal policy (episode outcome mean of
1) after 800K timesteps, whereas IPPO was only able to reach an episode outcome
mean of 0.966 after one million timesteps. Hyperparameter tuning greatly
improved training performance. Across one million timesteps the tuned
hyperparameters reached an optimal policy whereas the default hyperparameters
only managed to win sporadically, with most simulations resulting in a draw. We
tested a real-world constraint, attack detection alert success, and found that
when alert success probability is reduced to 0.75 or 0.9, the MARL defenders
were still able to win in over 97.5% or 99.5% of episodes, respectively.

该论文展示了自主网络防御在工业控制系统中的潜力，并提供了一个基线环境，进一步探索多智能体强化学习在此问题领域的应用。

多智能体强化学习在海上操作技术网络安全中的应用

Multi-Agent Reinforcement Learning for Maritime Operational Technology  Cyber Security

Recent advancements in deep learning techniques have opened new possibilities
for designing solutions for autonomous cyber defence. Teams of intelligent
agents in computer network defence roles may reveal promising avenues to
safeguard cyber and kinetic assets. In a simulated game environment, agents are
evaluated on their ability to jointly mitigate attacker activity in host-based
defence scenarios. Defender systems are evaluated against heuristic attackers
with the goals of compromising network confidentiality, integrity, and
availability. Value-based Independent Learning and Centralized Training
Decentralized Execution (CTDE) cooperative Multi-Agent Reinforcement Learning
(MARL) methods are compared revealing that both approaches outperform a simple
multi-agent heuristic defender. This work demonstrates the ability of
cooperative MARL to learn effective cyber defence tactics against varied
threats.

深度学习技术的最新进展为自主网络防御的设计提供了新的可能性，智能代理团队在计算机网络防御角色中可能揭示了保护网络和运动资产的有希望的途径，该研究对比了基于价值的独立学习和集中训练去中心化执行的合作多代理强化学习方法，表明这两种方法都优于简单的多代理启发式防御者，这项工作展示了合作多代理强化学习在学习有效的网络防御策略对抗不同威胁方面的能力。