There is a growing need to gain insight into language model capabilities that
relate to sensitive topics, such as bioterrorism or cyberwarfare. However,
traditional open source benchmarks are not fit for the task, due to the
associated practice of publishing the correct answers in human-readable form.
At the same time, enforcing mandatory closed-quarters evaluations might stifle
development and erode trust. In this context, we propose hashmarking, a
protocol for evaluating language models in the open without having to disclose
the correct answers. In its simplest form, a hashmark is a benchmark whose
reference solutions have been cryptographically hashed prior to publication.
Following an overview of the proposed evaluation protocol, we go on to assess
its resilience against traditional attack vectors (e.g. rainbow table attacks),
as well as against failure modes unique to increasingly capable generative
models.

通过哈希标记协议，在不泄露正确答案的情况下，评估语言模型的能力，并对传统和生成模型的攻击进行了评估。

散列标记：高风险 AI 评估的隐私保护基准

Hashmarks: Privacy-Preserving Benchmarks for High-Stakes AI Evaluation

In this work we start walking the path to a new perspective for viewing
cyberwarfare scenarios, by introducing conceptual tools (a formal model) to
evaluate the costs of an attack, to describe the theater of operations,
targets, missions, actions, plans and assets involved in cyberwarfare attacks.
We also describe two applications of this model: autonomous planning leading to
automated penetration tests, and attack simulations, allowing a system
administrator to evaluate the vulnerabilities of his network.

本文使用概念工具（形式模型）评估攻击的成本，并描述参与网络攻击的操作目标、任务、行动、计划和资产。在此基础上，提出两种模型应用：自主规划和攻击模拟，用于自动化渗透测试和网络漏洞评估。