As AI systems become more advanced, companies and regulators will make
difficult decisions about whether it is safe to train and deploy them. To
prepare for these decisions, we investigate how developers could make a 'safety
case,' which is a structured rationale that AI systems are unlikely to cause a
catastrophe. We propose a framework for organizing a safety case and discuss
four categories of arguments to justify safety: total inability to cause a
catastrophe, sufficiently strong control measures, trustworthiness despite
capability to cause harm, and deference to credible AI advisors. We evaluate
concrete examples of arguments in each category and outline how arguments could
be combined to justify that AI systems are safe to deploy.

AI 系统的安全性是一个重要的考量，因此作者提出了一个结构化的理由框架，包含四个类别的论证，以证明 AI 系统在训练和部署过程中不太可能引发灾难，并提到控制措施的强度、即使可能造成伤害也是可信的、以及可信的 AI 顾问的权威等内容。

安全案例：为先进人工智能系统提供安全证明

Safety Cases: Justifying the Safety of Advanced AI Systems

Most reinforcement learning algorithms with formal regret guarantees assume
all mistakes are reversible and rely on essentially trying all possible
options. This approach leads to poor outcomes when some mistakes are
irreparable or even catastrophic. We propose a variant of the contextual bandit
problem where the goal is to minimize the chance of catastrophe. Specifically,
we assume that the payoff each round represents the chance of avoiding
catastrophe that round, and try to maximize the product of payoffs (the overall
chance of avoiding catastrophe). To give the agent some chance of success, we
allow a limited number of queries to a mentor and assume a Lipschitz continuous
payoff function. We present an algorithm whose regret and rate of querying the
mentor both approach 0 as the time horizon grows, assuming a continuous 1D
state space and a relatively "simple" payoff function. We also provide a
matching lower bound: without the simplicity assumption: any algorithm either
constantly asks for help or is nearly guaranteed to cause catastrophe. Finally,
we identify the key obstacle to generalizing our algorithm to a
multi-dimensional state space.

通过假设每个回合的付出代表避免灾难的机会，我们提出了一种上下文匹配问题的变体，目标是尽量减少灾难的可能性，进而通过最大化付出的乘积来尽量避免灾难的总体机会。我们提供了一个算法，可以在时间范围增长时使后悔和对导师提问的频率都趋近于 0，假设有一个连续的 1D 状态空间和相对简单的付出函数。同时，我们提供了一个匹配的下界：在没有简单假设的情况下，任何算法要么持续寻求帮助，要么几乎肯定会造成灾难。最后，我们确定了将我们的算法推广到多维状态空间的关键障碍。

通过寻求帮助避免连续空间中的灾难

Avoiding Catastrophe in Continuous Spaces by Asking for Help

This report examines what I see as the core argument for concern about
existential risk from misaligned artificial intelligence. I proceed in two
stages. First, I lay out a backdrop picture that informs such concern. On this
picture, intelligent agency is an extremely powerful force, and creating agents
much more intelligent than us is playing with fire -- especially given that if
their objectives are problematic, such agents would plausibly have instrumental
incentives to seek power over humans. Second, I formulate and evaluate a more
specific six-premise argument that creating agents of this kind will lead to
existential catastrophe by 2070. On this argument, by 2070: (1) it will become
possible and financially feasible to build relevantly powerful and agentic AI
systems; (2) there will be strong incentives to do so; (3) it will be much
harder to build aligned (and relevantly powerful/agentic) AI systems than to
build misaligned (and relevantly powerful/agentic) AI systems that are still
superficially attractive to deploy; (4) some such misaligned systems will seek
power over humans in high-impact ways; (5) this problem will scale to the full
disempowerment of humanity; and (6) such disempowerment will constitute an
existential catastrophe. I assign rough subjective credences to the premises in
this argument, and I end up with an overall estimate of ~5% that an existential
catastrophe of this kind will occur by 2070. (May 2022 update: since making
this report public in April 2021, my estimate here has gone up, and is now at
>10%.)

本文探讨了关于人工智能失控所带来的灾难性威胁的核心论点，论述了超级智能对人类的潜在生存危险，提出了深度学习中存在问题的因素，并预测到在 2070 年前此问题的发展将导致人类的终极灭亡。