Prominent AI experts have suggested that companies developing high-risk AI
systems should be required to show that such systems are safe before they can
be developed or deployed. The goal of this paper is to expand on this idea and
explore its implications for risk management. We argue that entities developing
or deploying high-risk AI systems should be required to present evidence of
affirmative safety: a proactive case that their activities keep risks below
acceptable thresholds. We begin the paper by highlighting global security risks
from AI that have been acknowledged by AI experts and world governments. Next,
we briefly describe principles of risk management from other high-risk fields
(e.g., nuclear safety). Then, we propose a risk management approach for
advanced AI in which model developers must provide evidence that their
activities keep certain risks below regulator-set thresholds. As a first step
toward understanding what affirmative safety cases should include, we
illustrate how certain kinds of technical evidence and operational evidence can
support an affirmative safety case. In the technical section, we discuss
behavioral evidence (evidence about model outputs), cognitive evidence
(evidence about model internals), and developmental evidence (evidence about
the training process). In the operational section, we offer examples of
organizational practices that could contribute to affirmative safety cases:
information security practices, safety culture, and emergency response
capacity. Finally, we briefly compare our approach to the NIST AI Risk
Management Framework. Overall, we hope our work contributes to ongoing
discussions about national and global security risks posed by AI and regulatory
approaches to address these risks.

AI 专家建议在高风险的人工智能系统开发或部署之前，公司应该要求展示这些系统的安全性。本文旨在扩展这个想法并探讨其对风险管理的影响。我们认为开发或部署高风险人工智能系统的实体应该提供积极安全的证据，并确保其活动将风险保持在可接受的阈值以下。