Large Language Models (LLMs) have emerged as a powerful approach for driving offensive penetration-testing tooling. This paper analyzes the methodology and benchmarking practices used for evaluating Large Language Model (LLM)-driven attacks, focusing on offensive uses of LLMs in cybersecurity. We review 16 research papers detailing 15 prototypes and their respective testbeds. We detail our findings and provide actionable recommendations for future research, emphasizing the importance of extending existing testbeds, creating baselines, and including comprehensive metrics and qualitative analysis. We also note the distinction between security research and practice, suggesting that CTF-based challenges may not fully represent real-world penetration testing scenarios.

本研究针对现有大型语言模型（LLM）驱动的网络安全进攻性渗透测试工具的评估方法和基准实践进行分析，填补了在网络安全领域应用LLM的研究空白。文章重点提出了一种改进的研究方法，包括扩展现有测试平台、创建基线以及完善定量和定性分析的指标。研究结果表明，现有的CTF挑战可能无法全面反映真实世界中的渗透测试场景。

基于大型语言模型的进攻性安全基准实践：测试平台、指标与实验设计