Large Language Models (LLMs) have the potential to enhance Agent-Based
Modeling by better representing complex interdependent cybersecurity systems,
improving cybersecurity threat modeling and risk management. However,
evaluating LLMs in this context is crucial for legal compliance and effective
application development. Existing LLM evaluation frameworks often overlook the
human factor and cognitive computing capabilities essential for interdependent
cybersecurity. To address this gap, I propose OllaBench, a novel evaluation
framework that assesses LLMs' accuracy, wastefulness, and consistency in
answering scenario-based information security compliance and non-compliance
questions. OllaBench is built on a foundation of 24 cognitive behavioral
theories and empirical evidence from 38 peer-reviewed papers. OllaBench was
used to evaluate 21 LLMs, including both open-weight and commercial models from
OpenAI, Anthropic, Google, Microsoft, Meta and so on. The results reveal that
while commercial LLMs have the highest overall accuracy scores, there is
significant room for improvement. Smaller low-resolution open-weight LLMs are
not far behind in performance, and there are significant differences in token
efficiency and consistency among the evaluated models. OllaBench provides a
user-friendly interface and supports a wide range of LLM platforms, making it a
valuable tool for researchers and solution developers in the field of
human-centric interdependent cybersecurity and beyond.

通过 OllaBench 评估框架，综合考虑人类因素和认知计算能力，评估了 21 种大型语言模型在情景信息安全合规和不合规问题中的准确性、浪费程度和一致性，结果显示商业模型的整体准确性最高，但仍有改进的空间，小型开放权重模型表现不逊，模型间的记号效率和一致性存在显著差异，OllaBench 为人类中心的相互依赖性网络安全及其他领域的研究人员和解决方案开发者提供了一个用户友好的接口和广泛的 LLM 平台支持。