Automated penetration testing (AutoPT) based on reinforcement learning (RL)
has proven its ability to improve the efficiency of vulnerability
identification in information systems. However, RL-based PT encounters several
challenges, including poor sampling efficiency, intricate reward specification,
and limited interpretability. To address these issues, we propose a
knowledge-informed AutoPT framework called DRLRM-PT, which leverages reward
machines (RMs) to encode domain knowledge as guidelines for training a PT
policy. In our study, we specifically focus on lateral movement as a PT case
study and formulate it as a partially observable Markov decision process
(POMDP) guided by RMs. We design two RMs based on the MITRE ATT\&CK knowledge
base for lateral movement. To solve the POMDP and optimize the PT policy, we
employ the deep Q-learning algorithm with RM (DQRM). The experimental results
demonstrate that the DQRM agent exhibits higher training efficiency in PT
compared to agents without knowledge embedding. Moreover, RMs encoding more
detailed domain knowledge demonstrated better PT performance compared to RMs
with simpler knowledge.

基于强化学习的自动化渗透测试（AutoPT）已被证明能够提高信息系统中漏洞识别的效率。然而，基于强化学习的 PT 面临着一些挑战，包括采样效率低、奖励规定复杂以及可解释性有限。为解决这些问题，我们提出了一种基于知识的 AutoPT 框架，称为 DRLRM-PT，它利用奖励机器（RMs）将领域知识编码为培训 PT 策略的指南。在我们的研究中，我们特别关注作为 PT 案例研究的横向移动，并将其表述为由 RMs 引导的部分可观察马尔可夫决策过程（POMDP）。我们基于 MITRE ATT&CK 知识库为横向移动设计了两个 RMs。为了解决 POMDP 并优化 PT 策略，我们采用了带有 RM 的深度 Q 学习算法（DQRM）。实验结果表明，与不具有知识嵌入的 Agent 相比，DQRM Agent 在渗透测试中表现出更高的训练效率。此外，编码了更详细领域知识的 RMs 与编码较简单知识的 RMs 相比，在渗透测试性能方面表现更好。